Oct 31 21:15:18.730859 ip-10-0-136-156 systemd[1]: Starting Kubernetes Kubelet... Oct 31 21:15:19.150133 ip-10-0-136-156 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:19.150133 ip-10-0-136-156 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 21:15:19.150133 ip-10-0-136-156 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:19.150133 ip-10-0-136-156 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:19.154008 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.153921 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.154008 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154008 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.154008 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154012 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154015 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154018 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154021 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154023 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154026 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154029 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154031 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154034 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154036 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154039 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154041 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154044 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154046 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154049 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154054 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154056 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154059 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154062 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154065 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.154093 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154067 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154070 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154072 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154075 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154077 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154080 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154083 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154085 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154088 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154091 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154093 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154097 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154099 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154102 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154107 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154111 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154114 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154117 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154120 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.154578 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154122 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154125 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154128 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154130 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154133 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154136 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154139 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154142 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154144 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154147 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154150 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154152 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154155 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154157 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154160 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154162 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154166 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154168 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154171 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154175 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.155047 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154177 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154180 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154183 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154186 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154188 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154191 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154193 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154196 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154198 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154200 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154203 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154205 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154208 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154210 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154213 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154216 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154218 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154221 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154224 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154227 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.155500 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154229 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154232 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154234 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154236 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154239 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154241 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154244 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154247 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154250 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154253 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154256 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154258 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154261 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154263 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154265 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154269 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154271 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154274 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154279 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.156013 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154283 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154286 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154289 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154291 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154294 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.154296 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155100 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155107 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155109 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155112 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155115 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155117 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155121 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155125 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155129 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155133 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155135 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155138 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.156449 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155141 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155144 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155147 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155150 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155153 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155155 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155158 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155160 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155163 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155165 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155168 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155171 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155174 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155177 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155179 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155182 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155185 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155188 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155190 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155193 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.156888 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155195 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155198 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155201 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155203 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155206 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155208 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155210 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155213 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155216 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155219 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155221 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155223 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155226 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155228 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155231 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155233 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155236 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155238 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155241 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155243 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.157355 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155246 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155248 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155251 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155254 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155257 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155260 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155262 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155265 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155267 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155270 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155272 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155275 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155278 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155281 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155283 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155285 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155288 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155290 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155293 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155295 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.157827 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155299 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155302 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155304 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155307 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155309 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155312 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155315 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155318 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155320 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155323 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155325 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155328 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155331 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155333 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155336 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155339 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155341 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155344 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155347 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.158305 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155349 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155352 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155354 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155357 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155359 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155362 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155364 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155367 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155370 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155375 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155378 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155381 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155384 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155386 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155390 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155864 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155869 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155873 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155876 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.158800 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155878 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155881 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155885 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155889 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155891 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155894 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155898 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155901 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155903 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155906 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155911 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155914 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155917 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155920 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155922 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155925 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155927 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155930 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155933 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.159269 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155935 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155938 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155940 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155943 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155945 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155948 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155950 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155953 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155956 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155959 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155961 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155964 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155967 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155970 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155974 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155976 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155979 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155981 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155984 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.159730 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155986 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155989 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155992 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.155995 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156000 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156002 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156005 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156007 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156010 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156012 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156015 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156018 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156020 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156023 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156025 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156028 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156031 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156033 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156035 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156038 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.160166 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156041 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156043 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156046 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156048 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156051 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156053 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156056 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156059 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156062 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156064 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156067 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156069 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156072 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156074 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156076 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156079 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156082 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156085 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156087 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156090 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.160641 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156092 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156095 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156098 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156100 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156103 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156105 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156107 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156110 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156112 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156115 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156117 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156119 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156122 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156124 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156127 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156130 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156132 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156134 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156137 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.161101 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156140 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156143 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156145 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156148 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.156150 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157142 2582 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157158 2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157165 2582 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157170 2582 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157175 2582 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157179 2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157183 2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157188 2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157191 2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157194 2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157197 2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157201 2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157204 2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157207 2582 flags.go:64] FLAG: --cgroup-root="" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157210 2582 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157213 2582 flags.go:64] FLAG: --client-ca-file="" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157216 2582 flags.go:64] FLAG: --cloud-provider="external" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157218 2582 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 21:15:19.161586 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157232 2582 flags.go:64] FLAG: --cluster-domain="" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157236 2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157239 2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157242 2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157246 2582 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157255 2582 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157259 2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157262 2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157266 2582 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157269 2582 flags.go:64] FLAG: --contention-profiling="false" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157272 2582 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157275 2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157278 2582 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157282 2582 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157287 2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157290 2582 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157293 2582 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157296 2582 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157299 2582 flags.go:64] FLAG: --enable-server="true" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157302 2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157306 2582 flags.go:64] FLAG: --event-burst="100" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157309 2582 flags.go:64] FLAG: --event-qps="50" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157312 2582 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157315 2582 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 21:15:19.162104 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157318 2582 flags.go:64] FLAG: --eviction-hard="" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157323 2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157326 2582 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157329 2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157332 2582 flags.go:64] FLAG: --eviction-soft="" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157335 2582 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157338 2582 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157341 2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157344 2582 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157347 2582 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157349 2582 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157353 2582 flags.go:64] FLAG: --feature-gates="" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157358 2582 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157361 2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157364 2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157368 2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157371 2582 flags.go:64] FLAG: --healthz-port="10248" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157374 2582 flags.go:64] FLAG: --help="false" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157378 2582 flags.go:64] FLAG: --hostname-override="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157381 2582 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157384 2582 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157387 2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157390 2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 21:15:19.162725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157393 2582 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157396 2582 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157399 2582 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157402 2582 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157405 2582 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157408 2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157411 2582 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157413 2582 flags.go:64] FLAG: --kube-reserved="" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157416 2582 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157419 2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157422 2582 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157425 2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157428 2582 flags.go:64] FLAG: --lock-file="" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157431 2582 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157433 2582 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157436 2582 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157442 2582 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157445 2582 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157448 2582 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157451 2582 flags.go:64] FLAG: --logging-format="text" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157454 2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157457 2582 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157460 2582 flags.go:64] FLAG: --manifest-url="" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157463 2582 flags.go:64] FLAG: --manifest-url-header="" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157468 2582 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 21:15:19.163268 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157471 2582 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157475 2582 flags.go:64] FLAG: --max-pods="110" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157481 2582 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157484 2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157487 2582 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157490 2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157492 2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157495 2582 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157498 2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157505 2582 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157508 2582 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157511 2582 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157514 2582 flags.go:64] FLAG: --pod-cidr="" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157517 2582 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157520 2582 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157523 2582 flags.go:64] FLAG: --pods-per-core="0" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157526 2582 flags.go:64] FLAG: --port="10250" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157541 2582 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157546 2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-066a7b2ce8197dbe3" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157551 2582 flags.go:64] FLAG: --qos-reserved="" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157555 2582 flags.go:64] FLAG: --read-only-port="10255" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157558 2582 flags.go:64] FLAG: --register-node="true" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157561 2582 flags.go:64] FLAG: --register-with-taints="" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157565 2582 flags.go:64] FLAG: --registry-burst="10" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157568 2582 flags.go:64] FLAG: --registry-qps="5" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157570 2582 flags.go:64] FLAG: --reserved-cpus="" Oct 31 21:15:19.163880 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157573 2582 flags.go:64] FLAG: --reserved-memory="" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157577 2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157580 2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157584 2582 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157586 2582 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157589 2582 flags.go:64] FLAG: --runonce="false" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157592 2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157596 2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157601 2582 flags.go:64] FLAG: --seccomp-default="false" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157604 2582 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157606 2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157609 2582 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157612 2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157615 2582 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157618 2582 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157621 2582 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157624 2582 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157627 2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157630 2582 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157633 2582 flags.go:64] FLAG: --system-cgroups="" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157636 2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157641 2582 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157644 2582 flags.go:64] FLAG: --tls-cert-file="" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157647 2582 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157651 2582 flags.go:64] FLAG: --tls-min-version="" Oct 31 21:15:19.164615 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157654 2582 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157657 2582 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157660 2582 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157663 2582 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157666 2582 flags.go:64] FLAG: --v="2" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157670 2582 flags.go:64] FLAG: --version="false" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157674 2582 flags.go:64] FLAG: --vmodule="" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157678 2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.157681 2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157818 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157822 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157826 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157828 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157832 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157834 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157839 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157842 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157845 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157848 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157850 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.165469 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157854 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157858 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157861 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157864 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157867 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157869 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157872 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157874 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157877 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157879 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157884 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157887 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157889 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157893 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157895 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157898 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157900 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157903 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157905 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157908 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.166066 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157911 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157913 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157916 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157919 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157921 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157924 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157926 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157930 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157933 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157936 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157938 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157941 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157944 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157946 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157949 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157951 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157954 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157956 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157959 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157961 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.166561 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157964 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157966 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157970 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157973 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157976 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157978 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157981 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157983 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157986 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157989 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157991 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157993 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157996 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.157999 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158001 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158004 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158006 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158009 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158011 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158016 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.167021 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158018 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158022 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158025 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158027 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158030 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158032 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158034 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158037 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158039 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158041 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158044 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158046 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158049 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158051 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158055 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158058 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158060 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158063 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158065 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.167476 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158068 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158071 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158073 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158094 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158099 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158102 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158105 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158108 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158110 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158113 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158116 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158118 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158122 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158124 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158127 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158130 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.167958 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.158156 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158191 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158196 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158200 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158203 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158205 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158208 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158211 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158213 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158216 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158219 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158221 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158225 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158229 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158232 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158235 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158238 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158241 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158244 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.168333 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158247 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158249 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158252 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158254 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158257 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158259 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158262 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158264 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158267 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158271 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158274 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158276 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158279 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158282 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158285 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158287 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158290 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158293 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158295 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.168782 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158298 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158308 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158311 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158314 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158317 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158319 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158324 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158326 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158329 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158332 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158334 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158337 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158339 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158342 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158345 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158347 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158349 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158352 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158355 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158357 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.169221 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158360 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158362 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158366 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158368 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158371 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158373 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158376 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158379 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158382 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158384 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158387 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158389 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158392 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158394 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158397 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158399 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158402 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158404 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158408 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158411 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.169698 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158413 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158416 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158418 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158421 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158423 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158426 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158428 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158431 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158433 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158436 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158438 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158441 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158443 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158446 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158448 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158451 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158453 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158456 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158458 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.170157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158461 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158464 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158466 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158469 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158471 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158474 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158476 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158479 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158482 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.158485 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.166813 2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.166829 2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.167581 2582 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.167592 2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167653 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167658 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167661 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167664 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167667 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167670 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.170613 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167672 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167675 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167678 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167680 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167683 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167686 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167689 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167691 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167694 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167697 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167699 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167702 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167704 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167707 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167710 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167712 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167715 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167717 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167720 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167723 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.171108 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167726 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167729 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167735 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167739 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167742 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167745 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167747 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167750 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167752 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167755 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167758 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167761 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167763 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167766 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167769 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167771 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167774 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167776 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167781 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167784 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.171682 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167787 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167790 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167792 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167795 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167798 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167801 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167805 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167809 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167812 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167815 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167818 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167820 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167823 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167826 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167829 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167832 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167834 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167837 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167840 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.172132 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167842 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167845 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167848 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167850 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167852 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167855 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167857 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167860 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167862 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167865 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167868 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167870 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167873 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167875 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167879 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167881 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167884 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167887 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167889 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.172701 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167892 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167894 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167896 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167899 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167901 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167904 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167906 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167909 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167912 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167914 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167917 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167920 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167922 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167925 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167927 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167930 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167932 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167935 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167937 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.173174 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167940 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167942 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.167945 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.167972 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168128 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168134 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168138 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168141 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168143 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168147 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168149 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168152 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168155 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168157 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168160 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168162 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.173656 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168165 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168167 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168169 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168172 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168175 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168177 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168181 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168184 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168186 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168189 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168191 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168194 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168196 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168199 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168201 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168204 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168206 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168209 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168211 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168214 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.174082 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168216 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168219 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168221 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168224 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168226 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168229 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168232 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168234 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168237 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168239 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168242 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168246 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168250 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168254 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168257 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168260 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168262 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168265 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168268 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.174555 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168270 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168273 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168275 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168278 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168280 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168283 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168285 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168288 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168290 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168293 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168295 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168298 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168300 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168302 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168305 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168308 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168310 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168312 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168315 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.175004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168317 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168321 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168323 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168326 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168328 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168331 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168333 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168336 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168339 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168342 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168345 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168348 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168351 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168354 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168356 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168359 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168361 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168363 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168366 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168368 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.175752 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168371 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168374 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168376 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168379 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168382 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168386 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168389 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168391 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168394 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168396 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168399 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168401 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168404 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168406 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168409 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.176350 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168412 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.168435 2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168468 2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168472 2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168475 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168478 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168481 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168483 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168486 2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168489 2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168491 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168494 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168497 2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168499 2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168502 2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168504 2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:19.176731 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168507 2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168509 2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168512 2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168514 2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168516 2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168519 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168521 2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168524 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168526 2582 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168543 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168547 2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168550 2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168552 2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168555 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168558 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168561 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168564 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168567 2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168569 2582 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168572 2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:19.177157 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168574 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168577 2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168580 2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168582 2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168585 2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168588 2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168590 2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168593 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168595 2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168599 2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168603 2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168606 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168608 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168611 2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168613 2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168616 2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168618 2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168621 2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168623 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:19.177663 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168626 2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168628 2582 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168630 2582 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168633 2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168635 2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168638 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168640 2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168643 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168645 2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168648 2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168651 2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168654 2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168657 2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168660 2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168662 2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168665 2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168667 2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168670 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168672 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:19.178287 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168675 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168677 2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168679 2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168682 2582 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168684 2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168687 2582 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168689 2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168692 2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168694 2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168697 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168699 2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168702 2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168704 2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168707 2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168709 2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168712 2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168714 2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168717 2582 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168719 2582 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168721 2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:19.178788 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168724 2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168727 2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168733 2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168736 2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168740 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168742 2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168745 2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168747 2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168750 2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168752 2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168755 2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168757 2582 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168760 2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:19.168762 2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.168817 2582 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.170856 2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.172192 2582 server.go:1014] "Starting client certificate rotation" Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.172312 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 21:15:19.179279 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.172359 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 21:15:19.179811 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.174219 2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:19.179811 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.175358 2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:19.184736 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.184717 2582 log.go:25] "Validated CRI v1 runtime API" Oct 31 21:15:19.188217 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.188205 2582 log.go:25] "Validated CRI v1 image API" Oct 31 21:15:19.189127 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.189096 2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 21:15:19.192034 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.192015 2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c647af16-fcf6-4fc7-98e8-7097de119276:/dev/nvme0n1p3 ea906db2-42d2-4f17-907e-f99f9c2a69e5:/dev/nvme0n1p4] Oct 31 21:15:19.192109 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.192033 2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 21:15:19.200243 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.200118 2582 manager.go:217] Machine: {Timestamp:2026-10-31 21:15:19.198670373 +0000 UTC m=+0.347085282 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200460 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2625145046cbe0ef9bb84e3c79ee44 SystemUUID:ec262514-5046-cbe0-ef9b-b84e3c79ee44 BootID:84b4f89a-cd52-483a-a800-a0f0bff09960 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c6:50:23:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c6:50:23:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:3e:3c:b8:71:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 21:15:19.200243 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.200233 2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 21:15:19.200393 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.200321 2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 21:15:19.202736 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.202704 2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 21:15:19.202877 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.202738 2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 21:15:19.202915 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.202880 2582 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 21:15:19.202915 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.202887 2582 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 21:15:19.202963 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.202916 2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:19.202963 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.202932 2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:19.203807 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.203797 2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 21:15:19.207891 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.207876 2582 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 21:15:19.208007 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.207998 2582 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 21:15:19.208692 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.208683 2582 kubelet.go:493] "Attempting to sync node with API server" Oct 31 21:15:19.208729 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.208695 2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 21:15:19.208729 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.208706 2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 21:15:19.208729 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.208716 2582 kubelet.go:405] "Adding apiserver pod source" Oct 31 21:15:19.208729 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.208724 2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 21:15:19.209097 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.209076 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:19.211324 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.211309 2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 21:15:19.216649 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.216635 2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:19.217473 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217462 2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 21:15:19.217511 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217491 2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 21:15:19.217688 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217675 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 21:15:19.217736 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217698 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 21:15:19.217736 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217706 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 21:15:19.217736 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217715 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 21:15:19.217736 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217723 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 21:15:19.217847 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.217743 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 21:15:19.218393 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.218383 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 21:15:19.218422 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.218393 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 21:15:19.218422 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.218401 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 21:15:19.218422 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.218417 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 21:15:19.218497 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.218430 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 21:15:19.218497 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.218438 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 21:15:19.220134 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.220124 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 21:15:19.220134 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.220134 2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 21:15:19.223988 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.223974 2582 server.go:1263] "Started kubelet" Oct 31 21:15:19.224165 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.224073 2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 21:15:19.224225 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.224194 2582 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 21:15:19.224225 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.224212 2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 21:15:19.224811 ip-10-0-136-156 systemd[1]: Started Kubernetes Kubelet. Oct 31 21:15:19.226016 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.225256 2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 21:15:19.227382 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.226837 2582 server.go:317] "Adding debug handlers to kubelet server" Oct 31 21:15:19.229086 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.229056 2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 21:15:19.230476 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.230458 2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 21:15:19.230986 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.230966 2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 21:15:19.231117 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.231091 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:19.231598 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.231577 2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 21:15:19.231598 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.231579 2582 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 21:15:19.231738 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.231603 2582 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 21:15:19.231796 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.231745 2582 reconstruct.go:97] "Volume reconstruction finished" Oct 31 21:15:19.231796 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.231756 2582 reconciler.go:29] "Reconciler: start to sync state" Oct 31 21:15:19.231998 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.231984 2582 factory.go:55] Registering systemd factory Oct 31 21:15:19.232036 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.232000 2582 factory.go:223] Registration of the systemd container factory successfully Oct 31 21:15:19.232222 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.232212 2582 factory.go:153] Registering CRI-O factory Oct 31 21:15:19.232222 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.232223 2582 factory.go:223] Registration of the crio container factory successfully Oct 31 21:15:19.232294 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.232263 2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 21:15:19.232294 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.232279 2582 factory.go:103] Registering Raw factory Oct 31 21:15:19.232294 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.232289 2582 manager.go:1199] Started watching for new ooms in manager Oct 31 21:15:19.233208 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.233191 2582 manager.go:319] Starting recovery of all containers Oct 31 21:15:19.245470 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.245449 2582 manager.go:324] Recovery completed Oct 31 21:15:19.247948 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.247903 2582 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 31 21:15:19.250853 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.250838 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.252723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.252706 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.252790 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.252738 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.252790 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.252752 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.253209 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.253195 2582 cpu_manager.go:226] "Starting" policy="none" Oct 31 21:15:19.253209 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.253207 2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 21:15:19.253300 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.253223 2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 21:15:19.253300 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.253248 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.253908 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.253894 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.253968 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.253917 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.253968 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.253927 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.255452 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.255439 2582 policy_none.go:50] "Start" Oct 31 21:15:19.255507 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.255457 2582 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 21:15:19.255507 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.255467 2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 21:15:19.257407 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.257394 2582 policy_none.go:44] "Start" Oct 31 21:15:19.262961 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.262919 2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-156.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 21:15:19.275969 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.275943 2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 21:15:19.276501 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.275408 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18e3b9c1efa45fc9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-10-31 21:15:19.223943113 +0000 UTC m=+0.372358019,LastTimestamp:2026-10-31 21:15:19.223943113 +0000 UTC m=+0.372358019,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Oct 31 21:15:19.279261 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.279235 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-156.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 21:15:19.279360 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.279267 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 21:15:19.279360 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.279301 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.295304 2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.299613 2582 manager.go:341] "Starting Device Plugin manager" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.299658 2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.299668 2582 server.go:87] "Starting device plugin registration server" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.299918 2582 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.299929 2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.300089 2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.300179 2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.300187 2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.300662 2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.300714 2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.300820 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.302145 2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.302953 2582 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.302984 2582 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.303019 2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 21:15:19.313940 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.305372 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18e3b9c1f15b8434 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-10-31 21:15:19.25272274 +0000 UTC m=+0.401137645,LastTimestamp:2026-10-31 21:15:19.25272274 +0000 UTC m=+0.401137645,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Oct 31 21:15:19.325808 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.325785 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 21:15:19.327727 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.327649 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18e3b9c1f15bd72a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-10-31 21:15:19.252743978 +0000 UTC m=+0.401158883,LastTimestamp:2026-10-31 21:15:19.252743978 +0000 UTC m=+0.401158883,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Oct 31 21:15:19.353203 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.353176 2582 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Oct 31 21:15:19.353203 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.353110 2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18e3b9c1f15c058d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-10-31 21:15:19.252755853 +0000 UTC m=+0.401170758,LastTimestamp:2026-10-31 21:15:19.252755853 +0000 UTC m=+0.401170758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Oct 31 21:15:19.372084 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.371973 2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-156.ec2.internal.18e3b9c1f15b8434\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18e3b9c1f15b8434 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-10-31 21:15:19.25272274 +0000 UTC m=+0.401137645,LastTimestamp:2026-10-31 21:15:19.253907045 +0000 UTC m=+0.402321953,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Oct 31 21:15:19.400728 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.400655 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.402728 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.402706 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.402859 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.402739 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.402859 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.402756 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.402859 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.402783 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.403915 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.403897 2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal"] Oct 31 21:15:19.403990 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.403950 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.404760 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.404745 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.404824 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.404773 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.404824 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.404784 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.407070 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407056 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.407207 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407192 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.407253 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407223 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.407835 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407816 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.407835 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407827 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.407959 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407851 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.407959 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407861 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.407959 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407860 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.407959 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.407881 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.408939 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.408853 2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-156.ec2.internal.18e3b9c1f15bd72a\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18e3b9c1f15bd72a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-10-31 21:15:19.252743978 +0000 UTC m=+0.401158883,LastTimestamp:2026-10-31 21:15:19.253921912 +0000 UTC m=+0.402336819,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Oct 31 21:15:19.410048 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.410029 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.410134 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.410064 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.411186 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.411172 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.411259 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.411198 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.411259 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.411213 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.423978 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.423955 2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6ld86" Oct 31 21:15:19.432147 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.432127 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.432765 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.432743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.432852 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.432834 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48e3e5e26418af9437f262086f802d3b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"48e3e5e26418af9437f262086f802d3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.432899 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.432862 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.437556 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.437522 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6ld86" Oct 31 21:15:19.439022 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.439004 2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.453586 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.453492 2582 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-156.ec2.internal.18e3b9c1f15c058d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-156.ec2.internal.18e3b9c1f15c058d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-156.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-156.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-156.ec2.internal,},FirstTimestamp:2026-10-31 21:15:19.252755853 +0000 UTC m=+0.401170758,LastTimestamp:2026-10-31 21:15:19.253930576 +0000 UTC m=+0.402345481,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-156.ec2.internal,}" Oct 31 21:15:19.505787 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.505746 2582 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.524197 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.524169 2582 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-156.ec2.internal\" not found" node="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.533740 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.533713 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.533740 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.533744 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.533948 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.533760 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48e3e5e26418af9437f262086f802d3b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"48e3e5e26418af9437f262086f802d3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.533948 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.533812 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48e3e5e26418af9437f262086f802d3b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"48e3e5e26418af9437f262086f802d3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.533948 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.533824 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.533948 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.533824 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.706763 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.706675 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:19.707714 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.707697 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:19.707797 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.707729 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:19.707797 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.707743 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:19.707797 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.707783 2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.727978 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.727953 2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.727978 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.727980 2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-156.ec2.internal\": node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:19.735134 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.735116 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.742972 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:19.742955 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:19.768003 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.767974 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:19.868303 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:19.868268 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.049840 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.049617 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.150144 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.150112 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.250741 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.250712 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.252907 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.252886 2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 21:15:21.351680 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.351660 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.363654 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:21.363622 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb361f6202434cbe9c94eb392281c4180.slice/crio-4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39 WatchSource:0}: Error finding container 4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39: Status 404 returned error can't find the container with id 4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39 Oct 31 21:15:21.363845 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:21.363826 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e3e5e26418af9437f262086f802d3b.slice/crio-a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b WatchSource:0}: Error finding container a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b: Status 404 returned error can't find the container with id a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b Oct 31 21:15:21.368915 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.368873 2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:15:21.370122 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.370104 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:15:21.370233 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.370130 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:15:21.370233 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.370144 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:15:21.371362 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.371346 2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:15:21.373102 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.373078 2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:21.386541 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.386489 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"48e3e5e26418af9437f262086f802d3b","Type":"ContainerStarted","Data":"a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b"} Oct 31 21:15:21.387517 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.387495 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"b361f6202434cbe9c94eb392281c4180","Type":"ContainerStarted","Data":"4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39"} Oct 31 21:15:21.452138 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.452099 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.524701 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.524657 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 21:10:20 +0000 UTC" deadline="2028-05-09 09:03:31.391682235 +0000 UTC" Oct 31 21:15:21.524701 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.524694 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13331h48m9.866991522s" Oct 31 21:15:21.552848 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.552819 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.591650 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.591628 2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:21.653134 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.653071 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.753645 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.753603 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.854566 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.854512 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.955428 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:21.955357 2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:21.961725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.961695 2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:21.972721 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:21.972695 2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:15:22.012873 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.012691 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:15:22.033954 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.033921 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:22.035478 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.035440 2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:15:22.058877 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.058853 2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:15:22.291032 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.290942 2582 apiserver.go:52] "Watching apiserver" Oct 31 21:15:22.313153 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.313117 2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 21:15:22.374594 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.374559 2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 21:15:22.374758 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.374645 2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zb5bh" Oct 31 21:15:22.388317 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.388271 2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 21:15:22.388616 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.388589 2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-vbk79","openshift-ovn-kubernetes/ovnkube-node-zd6fn","kube-system/global-pull-secret-syncer-29psw","kube-system/konnectivity-agent-qwzj8","openshift-cluster-node-tuning-operator/tuned-jg7jt","openshift-network-diagnostics/network-check-target-z2f89","openshift-network-operator/iptables-alerter-9q252","kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd","openshift-image-registry/node-ca-2v76n","openshift-multus/network-metrics-daemon-mln5r","openshift-dns/node-resolver-74gq6","openshift-multus/multus-additional-cni-plugins-xdwqq"] Oct 31 21:15:22.391705 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.391141 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.393376 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.393343 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.395925 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.395899 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.396029 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.396005 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:22.398567 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.398399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.400557 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.400533 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.402532 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.402511 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:22.402644 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.402622 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:22.405093 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.404740 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.406933 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.406915 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.409274 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.409257 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.412173 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.412156 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.415128 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.415110 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.417044 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.417025 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:22.417143 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.417093 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:22.424295 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424271 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqxwz\" (UniqueName: \"kubernetes.io/projected/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-kube-api-access-pqxwz\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.424418 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424323 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-socket-dir-parent\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.424418 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424348 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-hostroot\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.424418 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424373 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-var-lib-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.424418 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424394 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-env-overrides\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.424418 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424408 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424422 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-system-cni-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424438 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cnibin\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424453 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424498 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-netns\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424529 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-bin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424565 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysconfig\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424605 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.424657 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424624 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424663 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-daemon-config\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424682 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-serviceca\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424696 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-run\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424712 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-netd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424743 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-script-lib\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-system-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424779 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-systemd\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424798 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/751552d9-639f-42f2-bdcb-06fd195d7c2e-agent-certs\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424819 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-multus\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424841 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fea989-b58f-42e5-b17a-a0d35ab2161a-host-slash\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424869 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mx2\" (UniqueName: \"kubernetes.io/projected/28fea989-b58f-42e5-b17a-a0d35ab2161a-kube-api-access-74mx2\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424897 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-conf\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424960 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-tmp\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.424985 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425004 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.425019 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425024 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425039 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-lib-modules\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425077 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovn-node-metrics-cert\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425093 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-device-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425107 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fea989-b58f-42e5-b17a-a0d35ab2161a-iptables-alerter-script\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-systemd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425139 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-bin\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-config\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425212 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425239 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-kubelet\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425257 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-slash\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425351 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbzgs\" (UniqueName: \"kubernetes.io/projected/311ad56a-c14b-43a5-8ada-505326087f8f-kube-api-access-nbzgs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425380 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-etc-kubernetes\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425400 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-host\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425424 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-var-lib-kubelet\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.425745 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425441 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425475 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/751552d9-639f-42f2-bdcb-06fd195d7c2e-konnectivity-ca\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425496 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cni-binary-copy\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425517 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzc8n\" (UniqueName: \"kubernetes.io/projected/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-kube-api-access-nzc8n\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425548 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-node-log\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425571 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-log-socket\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425594 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfmj\" (UniqueName: \"kubernetes.io/projected/cfc3bafa-8448-4f87-8480-f7311a31d1b4-kube-api-access-wvfmj\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425619 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425641 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-k8s-cni-cncf-io\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425666 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-multus-certs\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425725 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfa483f-5315-4500-8f1b-e42d50e9b31b-tmp-dir\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425742 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425756 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-modprobe-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425770 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-tuned\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425786 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-systemd-units\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425808 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-ovn\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.426247 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425826 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-registration-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425842 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxnsf\" (UniqueName: \"kubernetes.io/projected/bbfa483f-5315-4500-8f1b-e42d50e9b31b-kube-api-access-kxnsf\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425858 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-dbus\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425880 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-kubernetes\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425894 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-sys\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425943 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425964 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-socket-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425979 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-conf-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.425994 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bvs\" (UniqueName: \"kubernetes.io/projected/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-kube-api-access-75bvs\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426011 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-netns\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426030 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-etc-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426045 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f8jh\" (UniqueName: \"kubernetes.io/projected/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-kube-api-access-6f8jh\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426060 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfa483f-5315-4500-8f1b-e42d50e9b31b-hosts-file\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426078 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-host\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426100 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-sys-fs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426121 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cnibin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.427056 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426140 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-kubelet\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.427742 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.426162 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-kubelet-config\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.439947 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.439924 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 21:15:22.439947 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.439940 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:22.440101 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.439992 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 21:15:22.440101 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440031 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 21:15:22.440101 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440082 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 21:15:22.440248 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440108 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 21:15:22.440248 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440119 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 21:15:22.440248 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440128 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 21:15:22.440248 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440187 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 21:15:22.440248 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440211 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 21:15:22.440506 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440346 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:22.440506 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440408 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 21:15:22.440506 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440452 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 21:15:22.440648 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.440544 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 21:15:22.441095 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441074 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 21:15:22.441185 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441167 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 21:15:22.441242 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441210 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 21:15:22.441291 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441276 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 21:15:22.441405 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441368 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 21:15:22.441405 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441398 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 21:15:22.441515 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441277 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:15:22.441568 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441526 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdqcm\"" Oct 31 21:15:22.441568 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441565 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 21:15:22.441659 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441583 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vddbx\"" Oct 31 21:15:22.441659 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441604 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 21:15:22.441659 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441617 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 21:15:22.441659 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441648 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 21:15:22.441659 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441654 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9gmld\"" Oct 31 21:15:22.441883 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441696 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 21:15:22.441950 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.441919 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rdjnn\"" Oct 31 21:15:22.445059 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.445039 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g9h45\"" Oct 31 21:15:22.445159 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.445103 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 21:15:22.445159 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.445109 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p85m4\"" Oct 31 21:15:22.445159 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.445106 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xsvd6\"" Oct 31 21:15:22.445339 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.445179 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ddmwn\"" Oct 31 21:15:22.445339 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.445182 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6mmhq\"" Oct 31 21:15:22.513703 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.513667 2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 21:15:22.526538 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526500 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-netns\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526546 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-etc-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526575 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f8jh\" (UniqueName: \"kubernetes.io/projected/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-kube-api-access-6f8jh\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526603 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfa483f-5315-4500-8f1b-e42d50e9b31b-hosts-file\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526627 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-host\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526649 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-sys-fs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526652 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-netns\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526667 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cnibin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526713 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cnibin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.526723 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-kubelet\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526743 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-kubelet\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526740 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-kubelet-config\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526769 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxwz\" (UniqueName: \"kubernetes.io/projected/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-kube-api-access-pqxwz\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526786 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-socket-dir-parent\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526797 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfa483f-5315-4500-8f1b-e42d50e9b31b-hosts-file\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-hostroot\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526822 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-var-lib-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526825 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-etc-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526843 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-env-overrides\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526859 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-system-cni-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526887 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cnibin\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526900 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526913 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-netns\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526927 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-bin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526949 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysconfig\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.527053 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526962 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526976 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.526990 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-daemon-config\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527003 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-serviceca\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527021 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfr4n\" (UniqueName: \"kubernetes.io/projected/76b1019f-cd75-49a1-bc25-60e0788a8f73-kube-api-access-gfr4n\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527035 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-run\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527050 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-netd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527076 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-script-lib\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527091 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-system-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527106 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-systemd\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527120 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/751552d9-639f-42f2-bdcb-06fd195d7c2e-agent-certs\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527135 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-multus\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527149 2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527163 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fea989-b58f-42e5-b17a-a0d35ab2161a-host-slash\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527182 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mx2\" (UniqueName: \"kubernetes.io/projected/28fea989-b58f-42e5-b17a-a0d35ab2161a-kube-api-access-74mx2\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527196 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-conf\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527205 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.527624 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527237 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-netns\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527271 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-host\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527340 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-sys-fs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527374 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-kubelet-config\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527490 2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527594 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-socket-dir-parent\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527637 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-hostroot\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527681 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-var-lib-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527742 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-netd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527793 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-bin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527845 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysconfig\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527953 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527993 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.528211 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528171 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-env-overrides\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.528775 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528318 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.528775 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528364 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-system-cni-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.528775 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528405 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cnibin\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.528775 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.527209 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-tmp\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.528775 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528508 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-daemon-config\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.528775 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528522 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-system-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.529030 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528918 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-serviceca\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.529030 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528953 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-script-lib\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529030 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528967 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-systemd\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.529030 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.528992 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-conf\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.529030 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529002 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-multus\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.529030 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529014 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-run\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529033 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529055 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fea989-b58f-42e5-b17a-a0d35ab2161a-host-slash\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529059 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529100 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529111 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529127 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529156 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529177 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-lib-modules\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529200 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovn-node-metrics-cert\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529363 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529347 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529380 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-device-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529407 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fea989-b58f-42e5-b17a-a0d35ab2161a-iptables-alerter-script\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529428 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-systemd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529450 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-bin\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529474 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-config\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529499 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529524 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-kubelet\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529544 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-slash\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529567 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzgs\" (UniqueName: \"kubernetes.io/projected/311ad56a-c14b-43a5-8ada-505326087f8f-kube-api-access-nbzgs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529589 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-etc-kubernetes\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529586 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-bin\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529614 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-host\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-var-lib-kubelet\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529648 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-device-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.529662 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529663 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529687 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/751552d9-639f-42f2-bdcb-06fd195d7c2e-konnectivity-ca\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529711 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cni-binary-copy\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529734 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzc8n\" (UniqueName: \"kubernetes.io/projected/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-kube-api-access-nzc8n\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529755 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-node-log\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529779 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-log-socket\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529803 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfmj\" (UniqueName: \"kubernetes.io/projected/cfc3bafa-8448-4f87-8480-f7311a31d1b4-kube-api-access-wvfmj\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529839 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529862 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-k8s-cni-cncf-io\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529888 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-multus-certs\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529931 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfa483f-5315-4500-8f1b-e42d50e9b31b-tmp-dir\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529954 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.529978 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-modprobe-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530002 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-tuned\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530027 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-systemd-units\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.530054 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530053 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-ovn\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530079 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-registration-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530104 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxnsf\" (UniqueName: \"kubernetes.io/projected/bbfa483f-5315-4500-8f1b-e42d50e9b31b-kube-api-access-kxnsf\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530113 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fea989-b58f-42e5-b17a-a0d35ab2161a-iptables-alerter-script\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530128 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-dbus\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530152 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-kubernetes\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530162 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-systemd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530176 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-sys\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530201 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530226 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-socket-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530249 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-conf-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530272 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bvs\" (UniqueName: \"kubernetes.io/projected/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-kube-api-access-75bvs\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.530554 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530554 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-k8s-cni-cncf-io\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.531147 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530602 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-multus-certs\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.531147 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530621 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.531147 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530714 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-lib-modules\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.531147 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530759 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-var-lib-kubelet\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.531147 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530885 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-etc-kubernetes\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.531147 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530927 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-slash\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.531147 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.530976 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-kubelet\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.531478 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531229 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-host\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.531478 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531263 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-log-socket\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.531478 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.531329 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:22.531478 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531366 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-node-log\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.531478 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.531409 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret podName:bc8d0364-096a-49f7-87b3-ae72ae71f1a1 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:23.031390453 +0000 UTC m=+3.099036202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret") pod "global-pull-secret-syncer-29psw" (UID: "bc8d0364-096a-49f7-87b3-ae72ae71f1a1") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:22.531478 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531422 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-sys\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.531478 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531474 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-dbus\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531485 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-modprobe-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531519 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-kubernetes\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531550 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfa483f-5315-4500-8f1b-e42d50e9b31b-tmp-dir\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531609 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531616 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-socket-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531661 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531704 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-conf-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531745 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-ovn\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.531814 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531784 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-systemd-units\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.532226 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531832 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-registration-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.532226 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.531859 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/751552d9-639f-42f2-bdcb-06fd195d7c2e-konnectivity-ca\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.532226 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.532008 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cni-binary-copy\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.532226 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.532031 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-tmp\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.532481 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.532461 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-config\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.532539 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.532461 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.532770 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.532746 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/751552d9-639f-42f2-bdcb-06fd195d7c2e-agent-certs\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.533908 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.533890 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-tuned\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.535250 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.535223 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovn-node-metrics-cert\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.556195 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.555115 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:22.556195 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.555144 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:22.556195 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.555158 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m85cj for pod openshift-network-diagnostics/network-check-target-z2f89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:22.556195 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.555256 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj podName:8243464a-58b2-4d49-8004-72ad5bd9c085 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:23.055241741 +0000 UTC m=+3.122887484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m85cj" (UniqueName: "kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj") pod "network-check-target-z2f89" (UID: "8243464a-58b2-4d49-8004-72ad5bd9c085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:22.556524 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.556267 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f8jh\" (UniqueName: \"kubernetes.io/projected/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-kube-api-access-6f8jh\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.557949 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.557918 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bvs\" (UniqueName: \"kubernetes.io/projected/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-kube-api-access-75bvs\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.558410 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.558385 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mx2\" (UniqueName: \"kubernetes.io/projected/28fea989-b58f-42e5-b17a-a0d35ab2161a-kube-api-access-74mx2\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.558521 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.558417 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzc8n\" (UniqueName: \"kubernetes.io/projected/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-kube-api-access-nzc8n\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.559021 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.559004 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxnsf\" (UniqueName: \"kubernetes.io/projected/bbfa483f-5315-4500-8f1b-e42d50e9b31b-kube-api-access-kxnsf\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.559434 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.559415 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxwz\" (UniqueName: \"kubernetes.io/projected/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-kube-api-access-pqxwz\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.562003 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.561979 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfmj\" (UniqueName: \"kubernetes.io/projected/cfc3bafa-8448-4f87-8480-f7311a31d1b4-kube-api-access-wvfmj\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.563552 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.563533 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzgs\" (UniqueName: \"kubernetes.io/projected/311ad56a-c14b-43a5-8ada-505326087f8f-kube-api-access-nbzgs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.630901 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.630877 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:22.630994 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.630961 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr4n\" (UniqueName: \"kubernetes.io/projected/76b1019f-cd75-49a1-bc25-60e0788a8f73-kube-api-access-gfr4n\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:22.631042 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.631014 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:22.631091 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:22.631084 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs podName:76b1019f-cd75-49a1-bc25-60e0788a8f73 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:23.131062142 +0000 UTC m=+3.198707891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs") pod "network-metrics-daemon-mln5r" (UID: "76b1019f-cd75-49a1-bc25-60e0788a8f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:22.657911 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.657878 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr4n\" (UniqueName: \"kubernetes.io/projected/76b1019f-cd75-49a1-bc25-60e0788a8f73-kube-api-access-gfr4n\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:22.708220 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.708186 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:15:22.717995 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.717964 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-74gq6" Oct 31 21:15:22.719766 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.719736 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28fea989_b58f_42e5_b17a_a0d35ab2161a.slice/crio-2ada3ef530a19025ed68c760aa110c1cb40a5015913001a12d0857520ab80b62 WatchSource:0}: Error finding container 2ada3ef530a19025ed68c760aa110c1cb40a5015913001a12d0857520ab80b62: Status 404 returned error can't find the container with id 2ada3ef530a19025ed68c760aa110c1cb40a5015913001a12d0857520ab80b62 Oct 31 21:15:22.727401 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.727371 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:22.728710 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.728687 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfa483f_5315_4500_8f1b_e42d50e9b31b.slice/crio-286944098a4895be535194768b50bf46de7fcb04885487338350c516a6c82822 WatchSource:0}: Error finding container 286944098a4895be535194768b50bf46de7fcb04885487338350c516a6c82822: Status 404 returned error can't find the container with id 286944098a4895be535194768b50bf46de7fcb04885487338350c516a6c82822 Oct 31 21:15:22.730649 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.730631 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:15:22.739674 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.739641 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751552d9_639f_42f2_bdcb_06fd195d7c2e.slice/crio-b3c669d0178d0d21731a4f9e43a569cefaef5a20c1dcb72e3e37ece73851f517 WatchSource:0}: Error finding container b3c669d0178d0d21731a4f9e43a569cefaef5a20c1dcb72e3e37ece73851f517: Status 404 returned error can't find the container with id b3c669d0178d0d21731a4f9e43a569cefaef5a20c1dcb72e3e37ece73851f517 Oct 31 21:15:22.740596 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.740572 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a70fca7_fa0b_4b05_abbb_cb6258e84ae0.slice/crio-d34bc2b7dae381b00180fd9cb2bfd83445477595c57682dca0fc09c8d9504bed WatchSource:0}: Error finding container d34bc2b7dae381b00180fd9cb2bfd83445477595c57682dca0fc09c8d9504bed: Status 404 returned error can't find the container with id d34bc2b7dae381b00180fd9cb2bfd83445477595c57682dca0fc09c8d9504bed Oct 31 21:15:22.751529 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.751337 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbk79" Oct 31 21:15:22.757003 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.756979 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:15:22.763807 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.763772 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d7007d_ab88_47fb_a484_b6ee82e1f5ff.slice/crio-aba7215d5d886b0740917c1e4a7e1d9ab8e204801ddcd3a590b91eedef3da395 WatchSource:0}: Error finding container aba7215d5d886b0740917c1e4a7e1d9ab8e204801ddcd3a590b91eedef3da395: Status 404 returned error can't find the container with id aba7215d5d886b0740917c1e4a7e1d9ab8e204801ddcd3a590b91eedef3da395 Oct 31 21:15:22.768740 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.768711 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd778febb_a3d7_4ddc_a5de_cdcefe1e3952.slice/crio-4fbabdfd3f250ee18709eba8b1743ccd985d57083c3926e8a3ac174cf7a9dadf WatchSource:0}: Error finding container 4fbabdfd3f250ee18709eba8b1743ccd985d57083c3926e8a3ac174cf7a9dadf: Status 404 returned error can't find the container with id 4fbabdfd3f250ee18709eba8b1743ccd985d57083c3926e8a3ac174cf7a9dadf Oct 31 21:15:22.777171 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.777144 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:15:22.778697 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.778674 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:15:22.783164 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:22.783142 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:22.792556 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.792527 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af55054_ae66_45a2_a8ea_4a22b5e95bfe.slice/crio-587badc188b33cb56c3756445fc5308123abbb1ad188ccb1a599f77e0330c54e WatchSource:0}: Error finding container 587badc188b33cb56c3756445fc5308123abbb1ad188ccb1a599f77e0330c54e: Status 404 returned error can't find the container with id 587badc188b33cb56c3756445fc5308123abbb1ad188ccb1a599f77e0330c54e Oct 31 21:15:22.793004 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.792985 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311ad56a_c14b_43a5_8ada_505326087f8f.slice/crio-2f5ae43d5f5703a2b1bc2bd9b17b1ac7ee6ce147fdd2eab21fded2f2485b3be3 WatchSource:0}: Error finding container 2f5ae43d5f5703a2b1bc2bd9b17b1ac7ee6ce147fdd2eab21fded2f2485b3be3: Status 404 returned error can't find the container with id 2f5ae43d5f5703a2b1bc2bd9b17b1ac7ee6ce147fdd2eab21fded2f2485b3be3 Oct 31 21:15:22.797243 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:22.796480 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc3bafa_8448_4f87_8480_f7311a31d1b4.slice/crio-aff31a177faa6d685d8f31540cadde233944b438bf0cec8523efb2948ccc29c5 WatchSource:0}: Error finding container aff31a177faa6d685d8f31540cadde233944b438bf0cec8523efb2948ccc29c5: Status 404 returned error can't find the container with id aff31a177faa6d685d8f31540cadde233944b438bf0cec8523efb2948ccc29c5 Oct 31 21:15:23.033883 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.033847 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:23.034060 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.033995 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:23.034103 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.034078 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret podName:bc8d0364-096a-49f7-87b3-ae72ae71f1a1 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:24.034060752 +0000 UTC m=+4.101706491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret") pod "global-pull-secret-syncer-29psw" (UID: "bc8d0364-096a-49f7-87b3-ae72ae71f1a1") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:23.134651 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.134555 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:23.134816 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.134657 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:23.134816 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.134770 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:23.134927 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.134826 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs podName:76b1019f-cd75-49a1-bc25-60e0788a8f73 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:24.134812448 +0000 UTC m=+4.202458179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs") pod "network-metrics-daemon-mln5r" (UID: "76b1019f-cd75-49a1-bc25-60e0788a8f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:23.135190 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.135162 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:23.135190 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.135188 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:23.135281 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.135200 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m85cj for pod openshift-network-diagnostics/network-check-target-z2f89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:23.135281 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:23.135242 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj podName:8243464a-58b2-4d49-8004-72ad5bd9c085 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:24.135230509 +0000 UTC m=+4.202876240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m85cj" (UniqueName: "kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj") pod "network-check-target-z2f89" (UID: "8243464a-58b2-4d49-8004-72ad5bd9c085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:23.378267 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.378100 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 21:10:22 +0000 UTC" deadline="2028-06-09 16:52:19.259803776 +0000 UTC" Oct 31 21:15:23.378267 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.378165 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14083h36m55.88164301s" Oct 31 21:15:23.403811 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.403551 2582 generic.go:356] "Generic (PLEG): container finished" podID="b361f6202434cbe9c94eb392281c4180" containerID="9e3224391731f804c66f845c3cd0d96532e9a2ee0783a5b84320a1ef8625131e" exitCode=0 Oct 31 21:15:23.403811 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.403661 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"b361f6202434cbe9c94eb392281c4180","Type":"ContainerDied","Data":"9e3224391731f804c66f845c3cd0d96532e9a2ee0783a5b84320a1ef8625131e"} Oct 31 21:15:23.411133 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.411099 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" event={"ID":"311ad56a-c14b-43a5-8ada-505326087f8f","Type":"ContainerStarted","Data":"2f5ae43d5f5703a2b1bc2bd9b17b1ac7ee6ce147fdd2eab21fded2f2485b3be3"} Oct 31 21:15:23.418566 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.418463 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-74gq6" event={"ID":"bbfa483f-5315-4500-8f1b-e42d50e9b31b","Type":"ContainerStarted","Data":"286944098a4895be535194768b50bf46de7fcb04885487338350c516a6c82822"} Oct 31 21:15:23.431802 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.431746 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9q252" event={"ID":"28fea989-b58f-42e5-b17a-a0d35ab2161a","Type":"ContainerStarted","Data":"2ada3ef530a19025ed68c760aa110c1cb40a5015913001a12d0857520ab80b62"} Oct 31 21:15:23.452420 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.452379 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"48e3e5e26418af9437f262086f802d3b","Type":"ContainerStarted","Data":"b5ec9fe57064e5906d8c62e7fd300d56a68ddc58f125eccf0f2d2783c7f22ca5"} Oct 31 21:15:23.476367 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.476008 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"aff31a177faa6d685d8f31540cadde233944b438bf0cec8523efb2948ccc29c5"} Oct 31 21:15:23.481593 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.481491 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2v76n" event={"ID":"4af55054-ae66-45a2-a8ea-4a22b5e95bfe","Type":"ContainerStarted","Data":"587badc188b33cb56c3756445fc5308123abbb1ad188ccb1a599f77e0330c54e"} Oct 31 21:15:23.483064 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.482990 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerStarted","Data":"4fbabdfd3f250ee18709eba8b1743ccd985d57083c3926e8a3ac174cf7a9dadf"} Oct 31 21:15:23.489177 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.489147 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbk79" event={"ID":"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff","Type":"ContainerStarted","Data":"aba7215d5d886b0740917c1e4a7e1d9ab8e204801ddcd3a590b91eedef3da395"} Oct 31 21:15:23.493733 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.493685 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" podStartSLOduration=1.493668811 podStartE2EDuration="1.493668811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:23.493399657 +0000 UTC m=+3.561045413" watchObservedRunningTime="2026-10-31 21:15:23.493668811 +0000 UTC m=+3.561314566" Oct 31 21:15:23.495253 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.495226 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" event={"ID":"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0","Type":"ContainerStarted","Data":"d34bc2b7dae381b00180fd9cb2bfd83445477595c57682dca0fc09c8d9504bed"} Oct 31 21:15:23.515318 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:23.515270 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwzj8" event={"ID":"751552d9-639f-42f2-bdcb-06fd195d7c2e","Type":"ContainerStarted","Data":"b3c669d0178d0d21731a4f9e43a569cefaef5a20c1dcb72e3e37ece73851f517"} Oct 31 21:15:24.049539 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.048965 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:24.049539 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.049123 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:24.049539 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.049182 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret podName:bc8d0364-096a-49f7-87b3-ae72ae71f1a1 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:26.049166412 +0000 UTC m=+6.116812161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret") pod "global-pull-secret-syncer-29psw" (UID: "bc8d0364-096a-49f7-87b3-ae72ae71f1a1") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.149443 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.149548 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.149677 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.149781 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs podName:76b1019f-cd75-49a1-bc25-60e0788a8f73 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:26.149754968 +0000 UTC m=+6.217400704 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs") pod "network-metrics-daemon-mln5r" (UID: "76b1019f-cd75-49a1-bc25-60e0788a8f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.150187 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.150207 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.150219 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m85cj for pod openshift-network-diagnostics/network-check-target-z2f89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:24.150318 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.150265 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj podName:8243464a-58b2-4d49-8004-72ad5bd9c085 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:26.150250378 +0000 UTC m=+6.217896112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m85cj" (UniqueName: "kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj") pod "network-check-target-z2f89" (UID: "8243464a-58b2-4d49-8004-72ad5bd9c085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:24.378519 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.378421 2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 21:10:22 +0000 UTC" deadline="2028-03-30 16:56:01.947234967 +0000 UTC" Oct 31 21:15:24.378519 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.378452 2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12379h40m37.568786631s" Oct 31 21:15:24.389582 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.388810 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:24.389582 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.388935 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:24.389582 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.389253 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:24.389582 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.389359 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:24.389582 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.389421 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:24.389582 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:24.389493 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:24.530536 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:24.529186 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"b361f6202434cbe9c94eb392281c4180","Type":"ContainerStarted","Data":"72384640d3ff2538412c4b2387792e6fd7cbed3ab7e03448792ef8598936d37f"} Oct 31 21:15:25.537514 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:25.537477 2582 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="6b5c9c85367275497d6aa65dc3aa54e17f2f9c5c925ca2b7934b2427351897d2" exitCode=0 Oct 31 21:15:25.538582 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:25.538551 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerDied","Data":"6b5c9c85367275497d6aa65dc3aa54e17f2f9c5c925ca2b7934b2427351897d2"} Oct 31 21:15:25.602409 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:25.601969 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" podStartSLOduration=3.601953413 podStartE2EDuration="3.601953413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:24.572518587 +0000 UTC m=+4.640164342" watchObservedRunningTime="2026-10-31 21:15:25.601953413 +0000 UTC m=+5.669599167" Oct 31 21:15:26.064258 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:26.063662 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:26.064258 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.063839 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:26.064258 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.063907 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret podName:bc8d0364-096a-49f7-87b3-ae72ae71f1a1 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:30.063888934 +0000 UTC m=+10.131534669 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret") pod "global-pull-secret-syncer-29psw" (UID: "bc8d0364-096a-49f7-87b3-ae72ae71f1a1") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:26.164386 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:26.164490 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.164616 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.164676 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs podName:76b1019f-cd75-49a1-bc25-60e0788a8f73 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:30.164659404 +0000 UTC m=+10.232305148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs") pod "network-metrics-daemon-mln5r" (UID: "76b1019f-cd75-49a1-bc25-60e0788a8f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.165092 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.165110 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.165122 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m85cj for pod openshift-network-diagnostics/network-check-target-z2f89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:26.165246 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.165166 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj podName:8243464a-58b2-4d49-8004-72ad5bd9c085 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:30.165154422 +0000 UTC m=+10.232800153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m85cj" (UniqueName: "kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj") pod "network-check-target-z2f89" (UID: "8243464a-58b2-4d49-8004-72ad5bd9c085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:26.384341 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:26.384312 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:26.384619 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.384592 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:26.384953 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:26.384435 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:26.385061 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.385037 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:26.385895 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:26.385877 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:26.385990 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:26.385965 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:26.542150 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:26.542096 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qwzj8" event={"ID":"751552d9-639f-42f2-bdcb-06fd195d7c2e","Type":"ContainerStarted","Data":"7de9b718acadda0e955cc7ddf8fdb0db609a97e675e2d9108dc0d0744ca47faf"} Oct 31 21:15:27.327706 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:27.327669 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:27.328760 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:27.328734 2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:27.371030 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:27.370980 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qwzj8" podStartSLOduration=3.866005884 podStartE2EDuration="7.370961161s" totalImagesPullingTime="3.504955277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:26.585695099 +0000 UTC m=+6.653340853" watchObservedRunningTime="2026-10-31 21:15:27.370961161 +0000 UTC m=+7.438606915" Oct 31 21:15:28.164317 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:28.164120 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:28.165514 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:28.165090 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:15:28.388798 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:28.388677 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:28.388798 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:28.388711 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:28.389032 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:28.389000 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:28.391528 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:28.389115 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:28.391528 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:28.389221 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:28.391528 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:28.389458 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:28.550297 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:28.550263 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2v76n" event={"ID":"4af55054-ae66-45a2-a8ea-4a22b5e95bfe","Type":"ContainerStarted","Data":"661a3bb2931a3ae3a10e2ec9af44544b57562df78396d0d34e6674e31ae0428a"} Oct 31 21:15:29.556401 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:29.556355 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" event={"ID":"311ad56a-c14b-43a5-8ada-505326087f8f","Type":"ContainerStarted","Data":"c3efb291595765fda60adc99a46318f92c4ec2194ea158e73efee97c5b3f7e8f"} Oct 31 21:15:30.016974 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.016950 2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 21:15:30.101826 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.101270 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:30.101826 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.101432 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:30.101826 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.101492 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret podName:bc8d0364-096a-49f7-87b3-ae72ae71f1a1 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:38.1014756 +0000 UTC m=+18.169121345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret") pod "global-pull-secret-syncer-29psw" (UID: "bc8d0364-096a-49f7-87b3-ae72ae71f1a1") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:30.201729 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.201637 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:30.201729 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.201691 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:30.201883 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.201859 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:30.201945 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.201887 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:30.201945 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.201900 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m85cj for pod openshift-network-diagnostics/network-check-target-z2f89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:30.202044 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.201954 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj podName:8243464a-58b2-4d49-8004-72ad5bd9c085 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:38.201936383 +0000 UTC m=+18.269582119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m85cj" (UniqueName: "kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj") pod "network-check-target-z2f89" (UID: "8243464a-58b2-4d49-8004-72ad5bd9c085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:30.202044 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.201967 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:30.202044 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.202038 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs podName:76b1019f-cd75-49a1-bc25-60e0788a8f73 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:38.202012383 +0000 UTC m=+18.269658129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs") pod "network-metrics-daemon-mln5r" (UID: "76b1019f-cd75-49a1-bc25-60e0788a8f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:30.384629 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.384601 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:30.384820 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.384735 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:30.384820 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.384800 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:30.384921 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.384904 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:30.384963 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.384926 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:30.388381 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:30.385014 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:30.390472 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.389809 2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T21:15:30.016970081Z","UUID":"d8710709-bc78-43ac-bc43-387541dafcb9","Handler":null,"Name":"","Endpoint":""} Oct 31 21:15:30.391497 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.391478 2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 21:15:30.391497 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.391502 2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 21:15:30.560843 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.560135 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-74gq6" event={"ID":"bbfa483f-5315-4500-8f1b-e42d50e9b31b","Type":"ContainerStarted","Data":"97b5774211eb83b2d1064240f9c3dd9bd242a6ea18c607642c0f68dba387f212"} Oct 31 21:15:30.563685 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.563654 2582 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="ae3a86d8e960510ad9c6b904dbcc67ba77e456dcdd2d16c660e73c779e75bd07" exitCode=0 Oct 31 21:15:30.563819 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.563728 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerDied","Data":"ae3a86d8e960510ad9c6b904dbcc67ba77e456dcdd2d16c660e73c779e75bd07"} Oct 31 21:15:30.566906 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.566757 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbk79" event={"ID":"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff","Type":"ContainerStarted","Data":"f2d2ea4dd20fcd4a4a470071c6e035b3ee1c14c14e825970bd736238579932fe"} Oct 31 21:15:30.570461 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.570370 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" event={"ID":"311ad56a-c14b-43a5-8ada-505326087f8f","Type":"ContainerStarted","Data":"c300f27ab2ebb3b6acbc1cee0a67e7f2ebd5c454367ce13293d067796304f557"} Oct 31 21:15:30.603964 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.603907 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2v76n" podStartSLOduration=5.734292503 podStartE2EDuration="10.603890077s" totalImagesPullingTime="4.869597574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:28.593233586 +0000 UTC m=+8.660879341" watchObservedRunningTime="2026-10-31 21:15:30.603890077 +0000 UTC m=+10.671535831" Oct 31 21:15:30.604118 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:30.604060 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-74gq6" podStartSLOduration=3.3074831160000002 podStartE2EDuration="10.604053779s" totalImagesPullingTime="7.296570663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:30.60297325 +0000 UTC m=+10.670619003" watchObservedRunningTime="2026-10-31 21:15:30.604053779 +0000 UTC m=+10.671699534" Oct 31 21:15:31.575835 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:31.575611 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" event={"ID":"311ad56a-c14b-43a5-8ada-505326087f8f","Type":"ContainerStarted","Data":"d16ecb82497a74531205c309ab9627250677d5f18559c04981ec74ba2adbc681"} Oct 31 21:15:31.577992 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:31.577965 2582 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="be4c6a8986e381012171c927a7fadada0e2d9ebbf76f781e481549b9dfc13f5b" exitCode=0 Oct 31 21:15:31.578124 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:31.578082 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerDied","Data":"be4c6a8986e381012171c927a7fadada0e2d9ebbf76f781e481549b9dfc13f5b"} Oct 31 21:15:31.620873 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:31.620822 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" podStartSLOduration=3.912181348 podStartE2EDuration="11.620803665s" totalImagesPullingTime="7.708622317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:31.620780708 +0000 UTC m=+11.688426464" watchObservedRunningTime="2026-10-31 21:15:31.620803665 +0000 UTC m=+11.688449425" Oct 31 21:15:31.621260 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:31.621222 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-vbk79" podStartSLOduration=4.578073343 podStartE2EDuration="11.621213206s" totalImagesPullingTime="7.043139863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:30.698035055 +0000 UTC m=+10.765680810" watchObservedRunningTime="2026-10-31 21:15:31.621213206 +0000 UTC m=+11.688858965" Oct 31 21:15:32.384725 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.384657 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:32.384864 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:32.384765 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:32.384864 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.384669 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:32.384864 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:32.384840 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:32.384864 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.384858 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:32.385061 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:32.384895 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:32.582200 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.582162 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9q252" event={"ID":"28fea989-b58f-42e5-b17a-a0d35ab2161a","Type":"ContainerStarted","Data":"9584df9b7ed3e8eb193cc3c6e05814ff9e7e5259db4fe1695957ffcf8f324d5f"} Oct 31 21:15:32.585430 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.585402 2582 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="a96127d31bf2c33a079bf60f7eb15b78172137e3fec2f65899dfe5b6a18f73a1" exitCode=0 Oct 31 21:15:32.585571 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.585481 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerDied","Data":"a96127d31bf2c33a079bf60f7eb15b78172137e3fec2f65899dfe5b6a18f73a1"} Oct 31 21:15:32.586892 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.586868 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" event={"ID":"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0","Type":"ContainerStarted","Data":"29416e3a6bfad71bcbbe07e17ddf265ec9744f6accc9528218680c5a1128a69e"} Oct 31 21:15:32.624393 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.624347 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9q252" podStartSLOduration=5.149358498 podStartE2EDuration="12.624333061s" totalImagesPullingTime="7.474974563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:32.623907162 +0000 UTC m=+12.691552915" watchObservedRunningTime="2026-10-31 21:15:32.624333061 +0000 UTC m=+12.691978813" Oct 31 21:15:32.669517 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:32.669430 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" podStartSLOduration=3.296331555 podStartE2EDuration="12.669414781s" totalImagesPullingTime="9.373083226s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:32.668580632 +0000 UTC m=+12.736226384" watchObservedRunningTime="2026-10-31 21:15:32.669414781 +0000 UTC m=+12.737060535" Oct 31 21:15:34.387265 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:34.387239 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:34.387949 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:34.387363 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:34.387949 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:34.387435 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:34.387949 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:34.387536 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:34.387949 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:34.387572 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:34.387949 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:34.387644 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:35.596584 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:35.596351 2582 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="842b054604744e06b30d07d3d14819d319797490885a19b1acd2077befb113f0" exitCode=0 Oct 31 21:15:35.596584 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:35.596391 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerDied","Data":"842b054604744e06b30d07d3d14819d319797490885a19b1acd2077befb113f0"} Oct 31 21:15:36.384906 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:36.384875 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:36.384906 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:36.384895 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:36.385136 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:36.384938 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:36.385136 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:36.385042 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:36.385232 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:36.385152 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:36.385232 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:36.385208 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:36.600271 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:36.600237 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"67b85006c18d2c940cb6d41182df4fbbd65bb2b340605990f9dd9cb213fc48e7"} Oct 31 21:15:36.600635 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:36.600351 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"c3c285a5166bf73fa2a94f8460449aab05283ab03114249ef722d34962e1aaee"} Oct 31 21:15:36.603550 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:36.603527 2582 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="fa5252aa3bee5da4ced4a50a8cdccc561206badfcecc9d4bfdcc8486d6fccce0" exitCode=0 Oct 31 21:15:36.603651 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:36.603573 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerDied","Data":"fa5252aa3bee5da4ced4a50a8cdccc561206badfcecc9d4bfdcc8486d6fccce0"} Oct 31 21:15:37.607936 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.607762 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:15:37.608350 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.608201 2582 generic.go:356] "Generic (PLEG): container finished" podID="cfc3bafa-8448-4f87-8480-f7311a31d1b4" containerID="67b85006c18d2c940cb6d41182df4fbbd65bb2b340605990f9dd9cb213fc48e7" exitCode=1 Oct 31 21:15:37.608350 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.608271 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerDied","Data":"67b85006c18d2c940cb6d41182df4fbbd65bb2b340605990f9dd9cb213fc48e7"} Oct 31 21:15:37.608457 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.608402 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"2db5fe7b071da7665389957e610c63140ac49c91fe4006f365a0b1f6613cee8e"} Oct 31 21:15:37.608505 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.608456 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"30858b1884ac71812ffbe75f491e3efd618832e2943c821b7c2b1f9116597bf6"} Oct 31 21:15:37.608563 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.608507 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"35f62134a104a1a22491433a6dd58e75284a0af53a5003b4a935ca098c07face"} Oct 31 21:15:37.608617 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.608566 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"0031ddaf571905d7796a219f15d93c15e68a85f616a09c36ff55bb30d45f7766"} Oct 31 21:15:37.611023 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.610998 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" event={"ID":"d778febb-a3d7-4ddc-a5de-cdcefe1e3952","Type":"ContainerStarted","Data":"5d3897b9ebcf9a016cd39231ea4e1bd7022a8a1b3c1f248fbb3623d23e05206f"} Oct 31 21:15:37.687687 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:37.687650 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xdwqq" podStartSLOduration=7.688661496 podStartE2EDuration="17.687640521s" totalImagesPullingTime="9.998979025s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:37.687129569 +0000 UTC m=+17.754775321" watchObservedRunningTime="2026-10-31 21:15:37.687640521 +0000 UTC m=+17.755286274" Oct 31 21:15:38.165779 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:38.165748 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:38.165934 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.165854 2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:38.165934 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.165895 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret podName:bc8d0364-096a-49f7-87b3-ae72ae71f1a1 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:54.165884432 +0000 UTC m=+34.233530177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret") pod "global-pull-secret-syncer-29psw" (UID: "bc8d0364-096a-49f7-87b3-ae72ae71f1a1") : object "kube-system"/"original-pull-secret" not registered Oct 31 21:15:38.266907 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:38.266874 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:38.266907 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:38.266914 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:38.267038 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.267007 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:38.267038 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.267023 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 21:15:38.267038 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.267035 2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 21:15:38.267185 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.267042 2582 projected.go:196] Error preparing data for projected volume kube-api-access-m85cj for pod openshift-network-diagnostics/network-check-target-z2f89: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:38.267185 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.267060 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs podName:76b1019f-cd75-49a1-bc25-60e0788a8f73 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:54.267043432 +0000 UTC m=+34.334689177 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs") pod "network-metrics-daemon-mln5r" (UID: "76b1019f-cd75-49a1-bc25-60e0788a8f73") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 21:15:38.267185 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.267079 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj podName:8243464a-58b2-4d49-8004-72ad5bd9c085 nodeName:}" failed. No retries permitted until 2026-10-31 21:15:54.267069616 +0000 UTC m=+34.334715360 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m85cj" (UniqueName: "kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj") pod "network-check-target-z2f89" (UID: "8243464a-58b2-4d49-8004-72ad5bd9c085") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 21:15:38.384623 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:38.384598 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:38.384817 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:38.384626 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:38.384817 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:38.384683 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:38.384817 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.384759 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:38.384926 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.384833 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:38.384926 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:38.384890 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:39.617472 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:39.617443 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:15:39.617916 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:39.617781 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"41969eb852ff8ac0c2e1707ab121139be4f07081371079cd0ee33de83435782d"} Oct 31 21:15:40.384815 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:40.384781 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:40.384989 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:40.384868 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:40.384989 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:40.384916 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:40.385095 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:40.385015 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:40.385095 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:40.385034 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:40.385173 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:40.385150 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:41.625127 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.625096 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:15:41.625833 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.625431 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"94c8fe5fd12c68c74d39a75ddfe6455b267fc099b382d1dc17c13e065e0c0a69"} Oct 31 21:15:41.625833 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.625637 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:41.625833 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.625669 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:41.625960 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.625946 2582 scope.go:122] "RemoveContainer" containerID="67b85006c18d2c940cb6d41182df4fbbd65bb2b340605990f9dd9cb213fc48e7" Oct 31 21:15:41.642086 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.642067 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:41.642173 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.642168 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:41.691523 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:41.691504 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:15:42.384804 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:42.384767 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:42.384958 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:42.384781 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:42.384958 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:42.384883 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:42.384958 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:42.384914 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:42.385126 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:42.385031 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:42.385178 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:42.385119 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:42.632757 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:42.632733 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:15:42.633158 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:42.633128 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" event={"ID":"cfc3bafa-8448-4f87-8480-f7311a31d1b4","Type":"ContainerStarted","Data":"e0ea2d98a209af67c54d06f803b907a6e4ab4c9c8c0f325c6a234941231c0ab2"} Oct 31 21:15:42.704181 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:42.704126 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" podStartSLOduration=9.108211495 podStartE2EDuration="22.70410612s" totalImagesPullingTime="13.595894625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:42.70352071 +0000 UTC m=+22.771166465" watchObservedRunningTime="2026-10-31 21:15:42.70410612 +0000 UTC m=+22.771751875" Oct 31 21:15:43.024640 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:43.024607 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29psw"] Oct 31 21:15:43.024827 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:43.024707 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:43.024827 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:43.024808 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:43.027902 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:43.027874 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2f89"] Oct 31 21:15:43.028025 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:43.027957 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:43.028074 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:43.028035 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:43.028345 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:43.028295 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mln5r"] Oct 31 21:15:43.028408 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:43.028399 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:43.028481 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:43.028467 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:44.384870 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:44.384841 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:44.385365 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:44.384942 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:44.385365 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:44.385013 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:44.385365 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:44.385090 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:45.384533 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:45.384463 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:45.384643 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:45.384563 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:46.384091 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:46.384060 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:46.384558 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:46.384176 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:46.384558 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:46.384219 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:46.384558 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:46.384319 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:47.384419 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:47.384388 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:47.384924 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:47.384523 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:48.384771 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:48.384737 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:48.385159 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:48.384857 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-29psw" podUID="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" Oct 31 21:15:48.385159 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:48.384908 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:48.385159 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:48.384991 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z2f89" podUID="8243464a-58b2-4d49-8004-72ad5bd9c085" Oct 31 21:15:49.383922 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:49.383897 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:49.384067 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:49.384008 2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mln5r" podUID="76b1019f-cd75-49a1-bc25-60e0788a8f73" Oct 31 21:15:49.618966 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:49.618940 2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeReady" Oct 31 21:15:49.619344 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:49.619063 2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 21:15:50.384962 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:50.384925 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:50.385155 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:50.385068 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:50.416002 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:50.415957 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 21:15:50.416133 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:50.416015 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflcb\"" Oct 31 21:15:50.416133 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:50.416038 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j68t2\"" Oct 31 21:15:50.416133 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:50.416048 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 21:15:50.416133 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:50.416081 2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 21:15:51.384110 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:51.384072 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:51.403124 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:51.403095 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvv8l\"" Oct 31 21:15:51.403124 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:51.403104 2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 21:15:51.918925 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:51.918897 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-74gq6_bbfa483f-5315-4500-8f1b-e42d50e9b31b/dns-node-resolver/0.log" Oct 31 21:15:52.847534 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:52.847506 2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2v76n_4af55054-ae66-45a2-a8ea-4a22b5e95bfe/node-ca/0.log" Oct 31 21:15:54.183737 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.183702 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:54.187413 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.187383 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:54.284674 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.284643 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:15:54.284804 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.284690 2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:54.284804 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:54.284784 2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 21:15:54.284908 ip-10-0-136-156 kubenswrapper[2582]: E1031 21:15:54.284846 2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs podName:76b1019f-cd75-49a1-bc25-60e0788a8f73 nodeName:}" failed. No retries permitted until 2026-10-31 21:16:26.284830837 +0000 UTC m=+66.352476588 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs") pod "network-metrics-daemon-mln5r" (UID: "76b1019f-cd75-49a1-bc25-60e0788a8f73") : secret "metrics-daemon-secret" not found Oct 31 21:15:54.287046 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.287028 2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:54.359479 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.359457 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:15:54.376635 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.376613 2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:54.788343 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.788242 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-29psw"] Oct 31 21:15:54.790452 ip-10-0-136-156 kubenswrapper[2582]: W1031 21:15:54.790429 2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8d0364_096a_49f7_87b3_ae72ae71f1a1.slice/crio-76505cc4b8dc49fe36e042e849846d98a2f8dd4522d1c6555dd26b0906140f9c WatchSource:0}: Error finding container 76505cc4b8dc49fe36e042e849846d98a2f8dd4522d1c6555dd26b0906140f9c: Status 404 returned error can't find the container with id 76505cc4b8dc49fe36e042e849846d98a2f8dd4522d1c6555dd26b0906140f9c Oct 31 21:15:54.801647 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:54.799413 2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z2f89"] Oct 31 21:15:55.662007 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:55.661965 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2f89" event={"ID":"8243464a-58b2-4d49-8004-72ad5bd9c085","Type":"ContainerStarted","Data":"11b58fd7cc870d6eac88072f2808d27d2a83a0a3e1422922180da59dc693c9f9"} Oct 31 21:15:55.663271 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:55.663242 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29psw" event={"ID":"bc8d0364-096a-49f7-87b3-ae72ae71f1a1","Type":"ContainerStarted","Data":"76505cc4b8dc49fe36e042e849846d98a2f8dd4522d1c6555dd26b0906140f9c"} Oct 31 21:15:58.672969 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:58.672880 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z2f89" event={"ID":"8243464a-58b2-4d49-8004-72ad5bd9c085","Type":"ContainerStarted","Data":"6f2556a7cd385a5a3f82245e92febedf640f77a5adeae114f0bd3c07621b33a7"} Oct 31 21:15:58.673434 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:58.673024 2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:15:58.720697 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:58.720653 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z2f89" podStartSLOduration=35.255680721 podStartE2EDuration="38.720636457s" totalImagesPullingTime="3.464955736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:58.72049184 +0000 UTC m=+38.788137593" watchObservedRunningTime="2026-10-31 21:15:58.720636457 +0000 UTC m=+38.788282213" Oct 31 21:15:59.676590 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:59.676549 2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-29psw" event={"ID":"bc8d0364-096a-49f7-87b3-ae72ae71f1a1","Type":"ContainerStarted","Data":"e4c1a0257a20f3dd3294c3b7a9564a1672c6dddc3cc0f2f3cb6b1c5a0449c83f"} Oct 31 21:15:59.721235 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:59.721185 2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-29psw" podStartSLOduration=35.016215568 podStartE2EDuration="39.721172073s" totalImagesPullingTime="4.704956505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:15:59.720822743 +0000 UTC m=+39.788468498" watchObservedRunningTime="2026-10-31 21:15:59.721172073 +0000 UTC m=+39.788817827" Oct 31 21:15:59.780245 ip-10-0-136-156 kubenswrapper[2582]: I1031 21:15:59.780137 2582 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:59.780295 ip-10-0-136-156 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 21:15:59.798723 ip-10-0-136-156 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 21:15:59.798891 ip-10-0-136-156 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 21:15:59.798941 ip-10-0-136-156 systemd[1]: kubelet.service: Consumed 2.028s CPU time. Oct 31 21:15:59.809958 ip-10-0-136-156 systemd[1]: Starting Kubernetes Kubelet... Oct 31 21:15:59.904334 ip-10-0-136-156 kubenswrapper[4294]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:59.904334 ip-10-0-136-156 kubenswrapper[4294]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 21:15:59.904334 ip-10-0-136-156 kubenswrapper[4294]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:59.904334 ip-10-0-136-156 kubenswrapper[4294]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 21:15:59.906339 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906251 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.906339 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906340 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906344 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906349 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906354 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906358 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906361 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906364 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906366 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906369 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906372 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906374 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906377 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906380 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906382 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906385 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906389 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906392 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906395 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906397 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.906410 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906400 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906403 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906406 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906408 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906411 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906414 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906417 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906421 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906423 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906426 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906428 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906431 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906433 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906436 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906439 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906441 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906443 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906446 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906450 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906453 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.906912 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906455 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906458 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906461 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906463 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906466 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906468 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906470 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906473 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906475 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906479 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906481 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906484 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906486 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906490 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906495 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906498 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906501 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906504 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906507 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.907435 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906510 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906512 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906515 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906517 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906520 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906522 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906525 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906527 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906530 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906532 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906535 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906537 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906540 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906543 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906545 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906549 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906552 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906554 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906557 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906560 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.907899 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906562 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906565 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906568 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906570 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906573 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906575 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906578 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906580 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906583 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906586 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906589 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906592 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906595 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906597 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906600 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906602 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906605 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906607 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906610 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906613 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.908409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906615 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906618 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906620 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906623 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906626 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906629 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.906631 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907390 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907398 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907401 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907404 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907407 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907410 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907412 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907415 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907418 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907421 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907423 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907426 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907428 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.908909 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907432 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907435 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907438 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907441 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907444 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907447 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907449 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907452 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907455 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907457 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907460 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907462 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907465 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907468 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907470 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907473 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907475 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907478 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907480 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907483 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.909460 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907486 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907489 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907491 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907494 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907497 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907499 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907502 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907504 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907507 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907509 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907512 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907514 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907517 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907520 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907523 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907527 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907532 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907539 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907543 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.909948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907546 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907549 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907552 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907554 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907557 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907560 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907563 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907566 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907568 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907571 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907573 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907576 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907579 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907582 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907585 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907589 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907592 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907596 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907599 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.910432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907602 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907605 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907607 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907610 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907613 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907616 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907618 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907621 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907624 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907626 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907629 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907634 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907636 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907639 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907641 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907644 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907646 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907649 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907652 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907654 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.910911 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907657 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907659 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907662 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907665 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907668 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907670 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907673 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907675 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907678 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907680 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907683 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907687 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907689 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907692 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.907694 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908150 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908155 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908158 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908161 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908164 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.911433 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908166 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908169 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908171 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908177 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908179 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908182 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908185 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908187 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908190 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908193 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908195 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908198 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908200 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908203 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908206 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908209 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908212 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908214 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908217 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908220 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.911919 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908223 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908226 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908228 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908231 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908234 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908236 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908239 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908241 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908243 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908246 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908248 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908251 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908253 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908256 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908259 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908263 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908266 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908268 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908271 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908273 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.912555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908276 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908279 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908281 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908284 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908286 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908289 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908292 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908294 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908297 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908315 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908317 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908320 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908323 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908325 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908328 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908330 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908333 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908336 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908338 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908341 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.913422 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908344 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908347 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908349 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908352 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908355 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908357 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908360 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908364 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908368 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908373 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908376 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908379 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908383 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908386 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908389 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908391 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908394 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908396 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908399 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.914080 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908401 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908404 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908407 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908409 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908412 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908415 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908417 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908420 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908423 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908425 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908428 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908431 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908433 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908436 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908439 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908441 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908445 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908448 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908450 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.914553 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908453 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908456 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.908459 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908570 4294 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908587 4294 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908596 4294 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908604 4294 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908609 4294 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908612 4294 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908616 4294 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908620 4294 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908624 4294 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908626 4294 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908630 4294 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908633 4294 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908636 4294 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908639 4294 flags.go:64] FLAG: --cgroup-root="" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908642 4294 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908646 4294 flags.go:64] FLAG: --client-ca-file="" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908649 4294 flags.go:64] FLAG: --cloud-provider="external" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908652 4294 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908658 4294 flags.go:64] FLAG: --cluster-domain="" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908661 4294 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908664 4294 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 21:15:59.915031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908667 4294 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908671 4294 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908675 4294 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908678 4294 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908681 4294 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908684 4294 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908687 4294 flags.go:64] FLAG: --contention-profiling="false" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908690 4294 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908693 4294 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908696 4294 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908699 4294 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908703 4294 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908707 4294 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908710 4294 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908715 4294 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908718 4294 flags.go:64] FLAG: --enable-server="true" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908721 4294 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908726 4294 flags.go:64] FLAG: --event-burst="100" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908729 4294 flags.go:64] FLAG: --event-qps="50" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908732 4294 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908735 4294 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908738 4294 flags.go:64] FLAG: --eviction-hard="" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908742 4294 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908745 4294 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908748 4294 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 21:15:59.915678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908751 4294 flags.go:64] FLAG: --eviction-soft="" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908754 4294 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908757 4294 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908760 4294 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908764 4294 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908766 4294 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908769 4294 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908772 4294 flags.go:64] FLAG: --feature-gates="" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908777 4294 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908780 4294 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908783 4294 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908786 4294 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908789 4294 flags.go:64] FLAG: --healthz-port="10248" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908793 4294 flags.go:64] FLAG: --help="false" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908796 4294 flags.go:64] FLAG: --hostname-override="ip-10-0-136-156.ec2.internal" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908799 4294 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908802 4294 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908805 4294 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908808 4294 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908812 4294 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908815 4294 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908825 4294 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908828 4294 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 21:15:59.916318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908831 4294 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908834 4294 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908837 4294 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908840 4294 flags.go:64] FLAG: --kube-reserved="" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908843 4294 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908846 4294 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908850 4294 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908853 4294 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908856 4294 flags.go:64] FLAG: --lock-file="" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908859 4294 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908862 4294 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908865 4294 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908871 4294 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908874 4294 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908876 4294 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908879 4294 flags.go:64] FLAG: --logging-format="text" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908882 4294 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908885 4294 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908888 4294 flags.go:64] FLAG: --manifest-url="" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908892 4294 flags.go:64] FLAG: --manifest-url-header="" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908896 4294 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908899 4294 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908903 4294 flags.go:64] FLAG: --max-pods="110" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908906 4294 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908909 4294 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 21:15:59.916892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908912 4294 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908915 4294 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908918 4294 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908922 4294 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908925 4294 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908932 4294 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908939 4294 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908943 4294 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908946 4294 flags.go:64] FLAG: --pod-cidr="" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908949 4294 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908951 4294 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908954 4294 flags.go:64] FLAG: --pods-per-core="0" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908957 4294 flags.go:64] FLAG: --port="10250" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908962 4294 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908965 4294 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-066a7b2ce8197dbe3" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908968 4294 flags.go:64] FLAG: --qos-reserved="" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908971 4294 flags.go:64] FLAG: --read-only-port="10255" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908974 4294 flags.go:64] FLAG: --register-node="true" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908977 4294 flags.go:64] FLAG: --register-with-taints="" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908985 4294 flags.go:64] FLAG: --registry-burst="10" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908988 4294 flags.go:64] FLAG: --registry-qps="5" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908991 4294 flags.go:64] FLAG: --reserved-cpus="" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908993 4294 flags.go:64] FLAG: --reserved-memory="" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.908998 4294 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909001 4294 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909004 4294 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 21:15:59.917521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909006 4294 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909010 4294 flags.go:64] FLAG: --runonce="false" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909013 4294 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909016 4294 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909019 4294 flags.go:64] FLAG: --seccomp-default="false" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909021 4294 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909024 4294 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909027 4294 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909030 4294 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909033 4294 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909036 4294 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909039 4294 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909042 4294 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909046 4294 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909049 4294 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909052 4294 flags.go:64] FLAG: --system-cgroups="" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909055 4294 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909060 4294 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909063 4294 flags.go:64] FLAG: --tls-cert-file="" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909067 4294 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909074 4294 flags.go:64] FLAG: --tls-min-version="" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909081 4294 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909084 4294 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909087 4294 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909090 4294 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 21:15:59.918151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909093 4294 flags.go:64] FLAG: --v="2" Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909101 4294 flags.go:64] FLAG: --version="false" Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909105 4294 flags.go:64] FLAG: --vmodule="" Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909109 4294 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909112 4294 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909250 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909255 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909258 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909261 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909264 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909267 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909269 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909272 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909275 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909278 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909280 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909283 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909286 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909288 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909291 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.918825 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909293 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909312 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909316 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909320 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909322 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909325 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909329 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909332 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909334 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909337 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909339 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909342 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909344 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909347 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909349 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909352 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909354 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909357 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909359 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.919321 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909362 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909364 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909367 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909369 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909372 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909375 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909377 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909380 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909383 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909386 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909389 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909392 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909394 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909397 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909399 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909402 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909405 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909407 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909410 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909413 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.919891 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909416 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909419 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909421 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909424 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909426 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909429 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909431 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909434 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909437 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909439 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909442 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909446 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909449 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909451 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909454 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909456 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909459 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909462 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909465 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909468 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.920478 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909471 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909473 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909476 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909479 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909481 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909484 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909487 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909489 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909492 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909495 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909497 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909502 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909505 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909508 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909510 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909513 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909515 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909518 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909521 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909523 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.921071 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909526 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909528 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909531 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909535 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909537 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909540 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909543 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909546 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909548 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909551 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909555 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909558 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.909585 4294 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909618 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909622 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.921680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909625 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909628 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909630 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909633 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909636 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909638 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909641 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909644 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909648 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909651 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909654 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909656 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909659 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909661 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909664 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909667 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909669 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909672 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909674 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909677 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.922116 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909679 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909683 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909685 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909688 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909691 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909693 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909696 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909698 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909701 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909703 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909706 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909709 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909713 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909716 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909719 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909722 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909725 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909727 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909729 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.922728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909732 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909736 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909740 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909742 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909745 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909747 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909750 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909753 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909755 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909758 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909760 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909764 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909768 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909772 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909776 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909779 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909782 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909785 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909788 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.923259 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909790 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909792 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909795 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909798 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909800 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909803 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909805 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909808 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909811 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909813 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909816 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909818 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909821 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909823 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909829 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909831 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909834 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909837 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909839 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909842 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.923745 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909844 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909847 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909849 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909852 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909855 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909857 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909860 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909863 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909866 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909869 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909871 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909874 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909877 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909879 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909882 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909886 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909888 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909891 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909894 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909896 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.924242 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909899 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909902 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909904 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909907 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909909 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.909912 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.914911 4294 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.914929 4294 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.914951 4294 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.914956 4294 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915020 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915026 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915030 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915033 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915037 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915040 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915044 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915047 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915050 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915052 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.924776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915055 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915057 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915060 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915063 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915066 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915068 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915072 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915075 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915077 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915080 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915082 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915085 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915087 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915090 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915093 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915096 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915098 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915101 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915104 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915107 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.925265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915118 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915121 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915124 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915127 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915131 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915135 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915138 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915141 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915143 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915146 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915148 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915151 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915153 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915156 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915159 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915161 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915164 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915167 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915170 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.925765 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915173 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915175 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915178 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915180 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915183 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915186 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915188 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915192 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915194 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915197 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915200 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915203 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915205 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915208 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915211 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915214 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915217 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915219 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915222 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915224 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.926240 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915227 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915230 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915232 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915235 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915237 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915240 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915242 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915245 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915247 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915249 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915258 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915261 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915264 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915267 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915269 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915273 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915278 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915281 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915284 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.926800 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915288 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915291 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915294 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915297 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915319 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915322 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915325 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915328 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915330 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915333 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915336 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915338 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915341 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915344 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915347 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915349 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915352 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.927738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915354 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.915382 4294 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915546 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915551 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915554 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915557 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915560 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915568 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915571 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915574 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915576 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915580 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915582 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915585 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915587 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915590 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.928331 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915593 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915596 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915598 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915601 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915603 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915606 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915609 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915611 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915614 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915616 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915619 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915621 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915624 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915627 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915630 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915632 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915635 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915637 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915640 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915642 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.928728 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915645 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915647 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915650 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915652 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915656 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915659 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915661 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915664 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915667 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915669 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915672 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915674 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915677 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915680 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915683 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915685 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915688 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915691 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915695 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.929201 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915697 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915700 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915702 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915705 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915708 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915710 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915713 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915715 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915718 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915720 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915723 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915725 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915728 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915730 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915733 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915735 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915737 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915740 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915743 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915746 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.929664 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915748 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915751 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915754 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915757 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915760 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915763 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915765 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915768 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915770 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915775 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915778 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915780 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915783 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915785 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915789 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915793 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915796 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915799 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915801 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.930170 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915804 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915807 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915809 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915813 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915817 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915820 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915823 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915826 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915829 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915831 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915834 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915836 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915840 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915843 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.915867 4294 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 21:15:59.930716 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915898 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915902 4294 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915905 4294 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915907 4294 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915910 4294 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915913 4294 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915915 4294 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915919 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915922 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915925 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915929 4294 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915932 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915935 4294 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915938 4294 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915941 4294 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915943 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915946 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915949 4294 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915953 4294 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 21:15:59.931083 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915956 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915959 4294 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915962 4294 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915965 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915968 4294 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915972 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915974 4294 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915977 4294 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915980 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915982 4294 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915985 4294 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915988 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915991 4294 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915994 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.915997 4294 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916000 4294 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916002 4294 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916005 4294 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916007 4294 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916010 4294 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 21:15:59.931579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916014 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916016 4294 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916019 4294 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916021 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916024 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916026 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916029 4294 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916031 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916034 4294 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916036 4294 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916039 4294 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916041 4294 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916044 4294 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916046 4294 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916049 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916051 4294 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916054 4294 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916056 4294 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916059 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916061 4294 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 21:15:59.932073 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916064 4294 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916067 4294 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916069 4294 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916072 4294 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916075 4294 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916078 4294 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916080 4294 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916083 4294 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916085 4294 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916088 4294 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916090 4294 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916092 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916097 4294 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916099 4294 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916102 4294 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916104 4294 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916107 4294 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916109 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916111 4294 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916114 4294 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 21:15:59.932627 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916117 4294 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916119 4294 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916122 4294 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916124 4294 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916126 4294 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916129 4294 feature_gate.go:386] unrecognized feature gate: Example Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916131 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916134 4294 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916136 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916139 4294 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916142 4294 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916144 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916147 4294 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916149 4294 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916152 4294 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916155 4294 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916157 4294 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916160 4294 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916163 4294 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 21:15:59.933239 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916165 4294 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916168 4294 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916170 4294 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916173 4294 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916175 4294 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916179 4294 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916183 4294 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:15:59.916185 4294 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.916239 4294 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.917205 4294 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.917252 4294 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.917435 4294 server.go:1014] "Starting client certificate rotation" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.917575 4294 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.917627 4294 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 21:10:20 +0000 UTC" deadline="2028-07-31 08:53:53.937166426 +0000 UTC" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.917660 4294 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15323h37m54.019507922s" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.917857 4294 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:59.933828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.918868 4294 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 21:15:59.934229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.920279 4294 log.go:25] "Validated CRI v1 runtime API" Oct 31 21:15:59.934229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.921836 4294 log.go:25] "Validated CRI v1 image API" Oct 31 21:15:59.934229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.922563 4294 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 21:15:59.934229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.923929 4294 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c647af16-fcf6-4fc7-98e8-7097de119276:/dev/nvme0n1p3 ea906db2-42d2-4f17-907e-f99f9c2a69e5:/dev/nvme0n1p4] Oct 31 21:15:59.934366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.923948 4294 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11b58fd7cc870d6eac88072f2808d27d2a83a0a3e1422922180da59dc693c9f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11b58fd7cc870d6eac88072f2808d27d2a83a0a3e1422922180da59dc693c9f9/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/286944098a4895be535194768b50bf46de7fcb04885487338350c516a6c82822/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/286944098a4895be535194768b50bf46de7fcb04885487338350c516a6c82822/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2ada3ef530a19025ed68c760aa110c1cb40a5015913001a12d0857520ab80b62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2ada3ef530a19025ed68c760aa110c1cb40a5015913001a12d0857520ab80b62/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2f5ae43d5f5703a2b1bc2bd9b17b1ac7ee6ce147fdd2eab21fded2f2485b3be3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2f5ae43d5f5703a2b1bc2bd9b17b1ac7ee6ce147fdd2eab21fded2f2485b3be3/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fbabdfd3f250ee18709eba8b1743ccd985d57083c3926e8a3ac174cf7a9dadf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fbabdfd3f250ee18709eba8b1743ccd985d57083c3926e8a3ac174cf7a9dadf/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/587badc188b33cb56c3756445fc5308123abbb1ad188ccb1a599f77e0330c54e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/587badc188b33cb56c3756445fc5308123abbb1ad188ccb1a599f77e0330c54e/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76505cc4b8dc49fe36e042e849846d98a2f8dd4522d1c6555dd26b0906140f9c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76505cc4b8dc49fe36e042e849846d98a2f8dd4522d1c6555dd26b0906140f9c/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aba7215d5d886b0740917c1e4a7e1d9ab8e204801ddcd3a590b91eedef3da395/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aba7215d5d886b0740917c1e4a7e1d9ab8e204801ddcd3a590b91eedef3da395/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aff31a177faa6d685d8f31540cadde233944b438bf0cec8523efb2948ccc29c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aff31a177faa6d685d8f31540cadde233944b438bf0cec8523efb2948ccc29c5/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3c669d0178d0d21731a4f9e43a569cefaef5a20c1dcb72e3e37ece73851f517/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3c669d0178d0d21731a4f9e43a569cefaef5a20c1dcb72e3e37ece73851f517/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28fea989-b58f-42e5-b17a-a0d35ab2161a/volumes/kubernetes.io~projected/kube-api-access-74mx2:{mountpoint:/var/lib/kubelet/pods/28fea989-b58f-42e5-b17a-a0d35ab2161a/volumes/kubernetes.io~projected/kube-api-access-74mx2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/311ad56a-c14b-43a5-8ada-505326087f8f/volumes/kubernetes.io~projected/kube-api-access-nbzgs:{mountpoint:/var/lib/kubelet/pods/311ad56a-c14b-43a5-8ada-505326087f8f/volumes/kubernetes.io~projected/kube-api-access-nbzgs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4af55054-ae66-45a2-a8ea-4a22b5e95bfe/volumes/kubernetes.io~projected/kube-api-access-75bvs:{mountpoint:/var/lib/kubelet/pods/4af55054-ae66-45a2-a8ea-4a22b5e95bfe/volumes/kubernetes.io~projected/kube-api-access-75bvs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/751552d9-639f-42f2-bdcb-06fd195d7c2e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/751552d9-639f-42f2-bdcb-06fd195d7c2e/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76b1019f-cd75-49a1-bc25-60e0788a8f73/volumes/kubernetes.io~projected/kube-api-access-gfr4n:{mountpoint:/var/lib/kubelet/pods/76b1019f-cd75-49a1-bc25-60e0788a8f73/volumes/kubernetes.io~projected/kube-api-access-gfr4n major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8243464a-58b2-4d49-8004-72ad5bd9c085/volumes/kubernetes.io~projected/kube-api-access-m85cj:{mountpoint:/var/lib/kubelet/pods/8243464a-58b2-4d49-8004-72ad5bd9c085/volumes/kubernetes.io~projected/kube-api-access-m85cj major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~projected/kube-api-access-nzc8n:{mountpoint:/var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~projected/kube-api-access-nzc8n major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbfa483f-5315-4500-8f1b-e42d50e9b31b/volumes/kubernetes.io~projected/kube-api-access-kxnsf:{mountpoint:/var/lib/kubelet/pods/bbfa483f-5315-4500-8f1b-e42d50e9b31b/volumes/kubernetes.io~projected/kube-api-access-kxnsf major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc8d0364-096a-49f7-87b3-ae72ae71f1a1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/bc8d0364-096a-49f7-87b3-ae72ae71f1a1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volumes/kubernetes.io~projected/kube-api-access-wvfmj:{mountpoint:/var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volumes/kubernetes.io~projected/kube-api-access-wvfmj major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff/volumes/kubernetes.io~projected/kube-api-access-6f8jh:{mountpoint:/var/lib/kubelet/pods/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff/volumes/kubernetes.io~projected/kube-api-access-6f8jh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d778febb-a3d7-4ddc-a5de-cdcefe1e3952/volumes/kubernetes.io~projected/kube-api-access-pqxwz:{mountpoint:/var/lib/kubelet/pods/d778febb-a3d7-4ddc-a5de-cdcefe1e3952/volumes/kubernetes.io~projected/kube-api-access-pqxwz major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ae74b0f09eb2603bf1f1f0cd3c71bd789bb9502a4a548b456c000975b1e96f65/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a508e2d00869bdcda53ecfda51ff0fd7b109273db927757f8fb0be08a03e9b18/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/7c8eaed2e2b9c0b77b2f3845fbd2aa3b4b7c19ecf85e17702971a258680bd7cd/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b5e04151d4f918a191a51568a44380a5f0db2e5bef82f7b24fc833c12e44d77a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c3d9ab5b67c34845d7a8d653010b8636e7cfdd67a44c52236b1b1249279cbad2/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/aa380a9d2a8697b08f13038a4198efcfb98021d0345aed8e528d18e8690e3ad8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5e4080fec8d7be6594f69f48ad8a656cf8c3da060519b55d5a3d351eb159a285/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/8acf12a835b643e1972d4d787022ef02c5ff6ac18d69b56078b94f134c9ffe8c/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/2b54bceaa17ba542c0fc13137527a9dfaad6951a2c89ba61307fe57c3402cdb5/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/3c0e70c75eae58e49fdb8da2c0e554a5662edef7819c25bd77d655e07f71b32f/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/355ca48ce70565ba73768ff7c4065a1c964a6c4a6ff2488a7365e65e345e901d/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/78d088e6de3fd0387b4ee26a31f5dd7429ceb4d0806893118404dff4120c3d2d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/01dc98691460375aaedb46096725678e2611edd5596098f02a923c21b646ac4c/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-163:{mountpoint:/var/lib/containers/storage/overlay/33591d345e5e4123a918caa8639995abe027af32f592bb86180993e84b0ffdf9/merged major:0 minor:163 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/de58d874d6338b5cf0a6b8ff04dd8fdd631079d51f70b5b53e04d4152f19ec81/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/ba5bcd3a2a7c3b487285a05ef67622ec1235f73c63f5a1a01fa413ea4d4236a6/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a93f325cda76a632a7b20cd17571267bd943737265205ca476657b359adaa212/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b5e8a18ee7f3e9b146a32c18eea965421054c56c2559665f256929af290afcca/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cec8e35664470c7f4e89982051b6918b45d296b1e5e17b379556fa9d29348cbd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/2a29ae6ea03c078562fdd8a86553ffddb31000f8a4e904f0f514e6270d72a715/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/c9759f8a3214a52d8774b017d7eb7b571c667a5b69fca997085b10dc3d41bb87/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/6e02d24611473ab14c1bbdbbc077260594a0f3b7e54aa152ec3bba07e1642ef8/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/fffd922ec5695c6c3547112d33ac0ea3b2f5b5ac17c4c02a5616c523d7a49293/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8c67b60e5bad4d625fc676fc0c610a0bf5c3ad53aae6cd14772309251850522f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/f4fd9af539a17dfaf9661569ed4533de37154ca574ab8a1d9807a54871032bff/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/78ffb7faaa68beecf33763d7b15a11638724e25505512b2e888a19a732e02042/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-67:{mountpoint:/var/lib/containers/storage/overlay/44a6d38793e84de9b635af735140fe3dbfe35c93e130102b3b6683fafeee5b4c/merged major:0 minor:67 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/8db6a0beae1cf8975a26aea234b69d830090bcf84e81db33ae2554ea7500c15f/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/45cee5844ac357ecde7a7df2928e720f74f1f0ca3c2f82b43e5640d1c918b018/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/548d20fefd762409c8bc8a212269c6df0ef179cc443b55269e6b4b63328e1148/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/80554310d5158b570c8b85a7cdbbc1cf221d855bafe75eb8291fa969ee7a5db6/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/13c2922ddf9b8dddef08c984f60b8ab702185bba5c3f34e01208e55582cb975f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2a22ee86b16022333ba67772d4733f5ce1d20f6cc33471221197e52497bfada6/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/788936cdc01cd00aa362032dbfdd527cd94f20b21254143f769c9b4a21087abd/merged major:0 minor:96 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/5aa7111883105c514801a4499a09ee82b9523f89f220f751fd55971cb4a8c240/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.930973 4294 manager.go:217] Machine: {Timestamp:2026-10-31 21:15:59.930538012 +0000 UTC m=+0.055645235 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098149 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2625145046cbe0ef9bb84e3c79ee44 SystemUUID:ec262514-5046-cbe0-ef9b-b84e3c79ee44 BootID:84b4f89a-cd52-483a-a800-a0f0bff09960 Filesystems:[{Device:/var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/76b1019f-cd75-49a1-bc25-60e0788a8f73/volumes/kubernetes.io~projected/kube-api-access-gfr4n DeviceMajor:0 DeviceMinor:69 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-163 DeviceMajor:0 DeviceMinor:163 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/4af55054-ae66-45a2-a8ea-4a22b5e95bfe/volumes/kubernetes.io~projected/kube-api-access-75bvs DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/311ad56a-c14b-43a5-8ada-505326087f8f/volumes/kubernetes.io~projected/kube-api-access-nbzgs DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aba7215d5d886b0740917c1e4a7e1d9ab8e204801ddcd3a590b91eedef3da395/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bbfa483f-5315-4500-8f1b-e42d50e9b31b/volumes/kubernetes.io~projected/kube-api-access-kxnsf DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8243464a-58b2-4d49-8004-72ad5bd9c085/volumes/kubernetes.io~projected/kube-api-access-m85cj DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11b58fd7cc870d6eac88072f2808d27d2a83a0a3e1422922180da59dc693c9f9/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/587badc188b33cb56c3756445fc5308123abbb1ad188ccb1a599f77e0330c54e/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76505cc4b8dc49fe36e042e849846d98a2f8dd4522d1c6555dd26b0906140f9c/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc8d0364-096a-49f7-87b3-ae72ae71f1a1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volumes/kubernetes.io~projected/kube-api-access-wvfmj DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/751552d9-639f-42f2-bdcb-06fd195d7c2e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aff31a177faa6d685d8f31540cadde233944b438bf0cec8523efb2948ccc29c5/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-67 DeviceMajor:0 DeviceMinor:67 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/286944098a4895be535194768b50bf46de7fcb04885487338350c516a6c82822/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff/volumes/kubernetes.io~projected/kube-api-access-6f8jh DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d778febb-a3d7-4ddc-a5de-cdcefe1e3952/volumes/kubernetes.io~projected/kube-api-access-pqxwz DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/cfc3bafa-8448-4f87-8480-f7311a31d1b4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/28fea989-b58f-42e5-b17a-a0d35ab2161a/volumes/kubernetes.io~projected/kube-api-access-74mx2 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2ada3ef530a19025ed68c760aa110c1cb40a5015913001a12d0857520ab80b62/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0/volumes/kubernetes.io~projected/kube-api-access-nzc8n DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3c669d0178d0d21731a4f9e43a569cefaef5a20c1dcb72e3e37ece73851f517/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fbabdfd3f250ee18709eba8b1743ccd985d57083c3926e8a3ac174cf7a9dadf/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2f5ae43d5f5703a2b1bc2bd9b17b1ac7ee6ce147fdd2eab21fded2f2485b3be3/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:11b58fd7cc870d6 MacAddress:26:32:f7:c9:55:b2 Speed:10000 Mtu:8901} {Name:76505cc4b8dc49f MacAddress:1e:6d:ad:ac:2e:e1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c6:50:23:dd Speed:0 Mtu:9001} {Name:br-int MacAddress:96:da:b9:d0:22:a3 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c6:50:23:dd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0e:34:bf:b3:06:19 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:3e:3c:b8:71:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931215 4294 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931255 4294 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931510 4294 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931527 4294 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-156.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931642 4294 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931649 4294 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931659 4294 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931678 4294 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931757 4294 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931898 4294 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.931994 4294 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.932028 4294 kubelet.go:493] "Attempting to sync node with API server" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.932037 4294 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.932047 4294 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.932055 4294 kubelet.go:405] "Adding apiserver pod source" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.932062 4294 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.932839 4294 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.932854 4294 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:59.934486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.933372 4294 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 21:15:59.935384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935370 4294 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 21:15:59.935384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935382 4294 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 21:15:59.935494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935484 4294 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 21:15:59.935676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935666 4294 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 21:15:59.935706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935695 4294 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 21:15:59.935812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935803 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 21:15:59.935841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935819 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 21:15:59.935841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935827 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 21:15:59.935841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935837 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935848 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935860 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935873 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935881 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935889 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935896 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935916 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 21:15:59.935939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935933 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 21:15:59.936207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935967 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 21:15:59.936207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.935973 4294 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 21:15:59.939021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.939006 4294 server.go:1263] "Started kubelet" Oct 31 21:15:59.939553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.939133 4294 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 21:15:59.939553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.939289 4294 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 21:15:59.939553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.939267 4294 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 21:15:59.939722 ip-10-0-136-156 systemd[1]: Started Kubernetes Kubelet. Oct 31 21:15:59.942413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.942393 4294 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 21:15:59.943539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.943511 4294 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 21:15:59.943539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.943522 4294 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 21:15:59.943842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.943552 4294 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 21:15:59.943842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.940046 4294 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 21:15:59.943842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.943785 4294 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 21:15:59.943994 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:15:59.943851 4294 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-156.ec2.internal\" not found" Oct 31 21:15:59.944190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.944099 4294 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 21:10:22 +0000 UTC" deadline="2028-06-22 08:51:05.716083008 +0000 UTC" Oct 31 21:15:59.944266 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.944189 4294 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14387h35m5.77189633s" Oct 31 21:15:59.945666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945626 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfa483f-5315-4500-8f1b-e42d50e9b31b" volumeName="kubernetes.io/empty-dir/bbfa483f-5315-4500-8f1b-e42d50e9b31b-tmp-dir" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945667 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbfa483f-5315-4500-8f1b-e42d50e9b31b" volumeName="kubernetes.io/projected/bbfa483f-5315-4500-8f1b-e42d50e9b31b-kube-api-access-kxnsf" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945682 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc3bafa-8448-4f87-8480-f7311a31d1b4" volumeName="kubernetes.io/secret/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945694 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d7007d-ab88-47fb-a484-b6ee82e1f5ff" volumeName="kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cni-binary-copy" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945707 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" volumeName="kubernetes.io/projected/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-kube-api-access-pqxwz" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945718 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="751552d9-639f-42f2-bdcb-06fd195d7c2e" volumeName="kubernetes.io/configmap/751552d9-639f-42f2-bdcb-06fd195d7c2e-konnectivity-ca" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945729 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76b1019f-cd75-49a1-bc25-60e0788a8f73" volumeName="kubernetes.io/projected/76b1019f-cd75-49a1-bc25-60e0788a8f73-kube-api-access-gfr4n" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945742 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8243464a-58b2-4d49-8004-72ad5bd9c085" volumeName="kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj" seLinuxMountContext="" Oct 31 21:15:59.945757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945757 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a70fca7-fa0b-4b05-abbb-cb6258e84ae0" volumeName="kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-tmp" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945772 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a70fca7-fa0b-4b05-abbb-cb6258e84ae0" volumeName="kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-tuned" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945785 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc3bafa-8448-4f87-8480-f7311a31d1b4" volumeName="kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-script-lib" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945797 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d7007d-ab88-47fb-a484-b6ee82e1f5ff" volumeName="kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-daemon-config" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945807 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" volumeName="kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945821 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28fea989-b58f-42e5-b17a-a0d35ab2161a" volumeName="kubernetes.io/configmap/28fea989-b58f-42e5-b17a-a0d35ab2161a-iptables-alerter-script" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945837 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28fea989-b58f-42e5-b17a-a0d35ab2161a" volumeName="kubernetes.io/projected/28fea989-b58f-42e5-b17a-a0d35ab2161a-kube-api-access-74mx2" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945848 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af55054-ae66-45a2-a8ea-4a22b5e95bfe" volumeName="kubernetes.io/configmap/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-serviceca" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945860 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="751552d9-639f-42f2-bdcb-06fd195d7c2e" volumeName="kubernetes.io/secret/751552d9-639f-42f2-bdcb-06fd195d7c2e-agent-certs" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945871 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a70fca7-fa0b-4b05-abbb-cb6258e84ae0" volumeName="kubernetes.io/projected/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-kube-api-access-nzc8n" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945883 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc8d0364-096a-49f7-87b3-ae72ae71f1a1" volumeName="kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945894 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc3bafa-8448-4f87-8480-f7311a31d1b4" volumeName="kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-env-overrides" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945905 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc3bafa-8448-4f87-8480-f7311a31d1b4" volumeName="kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-config" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945917 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="311ad56a-c14b-43a5-8ada-505326087f8f" volumeName="kubernetes.io/projected/311ad56a-c14b-43a5-8ada-505326087f8f-kube-api-access-nbzgs" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945929 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfc3bafa-8448-4f87-8480-f7311a31d1b4" volumeName="kubernetes.io/projected/cfc3bafa-8448-4f87-8480-f7311a31d1b4-kube-api-access-wvfmj" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945942 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5d7007d-ab88-47fb-a484-b6ee82e1f5ff" volumeName="kubernetes.io/projected/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-kube-api-access-6f8jh" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945952 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" volumeName="kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.945989 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" volumeName="kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-binary-copy" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.946005 4294 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4af55054-ae66-45a2-a8ea-4a22b5e95bfe" volumeName="kubernetes.io/projected/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-kube-api-access-75bvs" seLinuxMountContext="" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.946017 4294 reconstruct.go:97] "Volume reconstruction finished" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.946024 4294 reconciler.go:29] "Reconciler: start to sync state" Oct 31 21:15:59.946133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.946053 4294 server.go:317] "Adding debug handlers to kubelet server" Oct 31 21:15:59.948045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948030 4294 factory.go:55] Registering systemd factory Oct 31 21:15:59.948045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948045 4294 factory.go:223] Registration of the systemd container factory successfully Oct 31 21:15:59.948516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948501 4294 factory.go:153] Registering CRI-O factory Oct 31 21:15:59.948570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948519 4294 factory.go:223] Registration of the crio container factory successfully Oct 31 21:15:59.948619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948572 4294 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 21:15:59.948619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948594 4294 factory.go:103] Registering Raw factory Oct 31 21:15:59.948619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948616 4294 manager.go:1199] Started watching for new ooms in manager Oct 31 21:15:59.948933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.948921 4294 manager.go:319] Starting recovery of all containers Oct 31 21:15:59.953716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.953698 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:15:59.953813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.953801 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:15:59.975780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.975740 4294 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 21:15:59.976862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.976839 4294 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 21:15:59.976985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.976873 4294 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 21:15:59.976985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.976895 4294 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 21:15:59.977137 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:15:59.977112 4294 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 21:15:59.987075 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.987039 4294 generic.go:356] "Generic (PLEG): container finished" podID="b361f6202434cbe9c94eb392281c4180" containerID="9e3224391731f804c66f845c3cd0d96532e9a2ee0783a5b84320a1ef8625131e" exitCode=0 Oct 31 21:15:59.993903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.993882 4294 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="fa5252aa3bee5da4ced4a50a8cdccc561206badfcecc9d4bfdcc8486d6fccce0" exitCode=0 Oct 31 21:15:59.993903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.993902 4294 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="842b054604744e06b30d07d3d14819d319797490885a19b1acd2077befb113f0" exitCode=0 Oct 31 21:15:59.994052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.993909 4294 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="a96127d31bf2c33a079bf60f7eb15b78172137e3fec2f65899dfe5b6a18f73a1" exitCode=0 Oct 31 21:15:59.994052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.993915 4294 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="be4c6a8986e381012171c927a7fadada0e2d9ebbf76f781e481549b9dfc13f5b" exitCode=0 Oct 31 21:15:59.994052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.993922 4294 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="ae3a86d8e960510ad9c6b904dbcc67ba77e456dcdd2d16c660e73c779e75bd07" exitCode=0 Oct 31 21:15:59.994052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.993930 4294 generic.go:356] "Generic (PLEG): container finished" podID="d778febb-a3d7-4ddc-a5de-cdcefe1e3952" containerID="6b5c9c85367275497d6aa65dc3aa54e17f2f9c5c925ca2b7934b2427351897d2" exitCode=0 Oct 31 21:15:59.995226 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.995206 4294 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 21:15:59.996783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:15:59.996761 4294 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:16:00.004572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.004553 4294 manager.go:324] Recovery completed Oct 31 21:16:00.005487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.005468 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:16:00.005809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.005789 4294 generic.go:356] "Generic (PLEG): container finished" podID="cfc3bafa-8448-4f87-8480-f7311a31d1b4" containerID="67b85006c18d2c940cb6d41182df4fbbd65bb2b340605990f9dd9cb213fc48e7" exitCode=1 Oct 31 21:16:00.007319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.007283 4294 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:16:00.009062 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.009041 4294 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:16:00.009062 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.009065 4294 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 21:16:00.018646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018628 4294 cpu_manager.go:226] "Starting" policy="none" Oct 31 21:16:00.018646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018640 4294 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 21:16:00.018773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018655 4294 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 21:16:00.018773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018764 4294 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 21:16:00.018860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018775 4294 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 21:16:00.018860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018789 4294 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 21:16:00.018860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018795 4294 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 21:16:00.018860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018800 4294 policy_none.go:50] "Start" Oct 31 21:16:00.018860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018823 4294 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 21:16:00.018860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018832 4294 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 21:16:00.019025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018962 4294 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 21:16:00.019025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018970 4294 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 21:16:00.019025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.018975 4294 policy_none.go:44] "Start" Oct 31 21:16:00.022741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.022718 4294 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 21:16:00.025234 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.025221 4294 manager.go:341] "Starting Device Plugin manager" Oct 31 21:16:00.025678 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:16:00.025248 4294 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 21:16:00.025748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.025681 4294 server.go:87] "Starting device plugin registration server" Oct 31 21:16:00.025957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.025943 4294 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 21:16:00.026017 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.025958 4294 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 21:16:00.026076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.026049 4294 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 21:16:00.026175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.026140 4294 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 21:16:00.026175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.026181 4294 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 21:16:00.026175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.026189 4294 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 21:16:00.026395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.026226 4294 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T21:16:00.026170403Z","UUID":"0c3dd303-2093-4b43-98cb-a9d7f4345064","Handler":null,"Name":"","Endpoint":""} Oct 31 21:16:00.027279 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:16:00.027257 4294 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 21:16:00.027616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.027595 4294 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 21:16:00.027715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.027635 4294 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 21:16:00.078102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078038 4294 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal"] Oct 31 21:16:00.078617 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078572 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"b361f6202434cbe9c94eb392281c4180","Type":"ContainerStarted","Data":"72384640d3ff2538412c4b2387792e6fd7cbed3ab7e03448792ef8598936d37f"} Oct 31 21:16:00.078617 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078613 4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.078731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078633 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"b361f6202434cbe9c94eb392281c4180","Type":"ContainerDied","Data":"9e3224391731f804c66f845c3cd0d96532e9a2ee0783a5b84320a1ef8625131e"} Oct 31 21:16:00.078731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078650 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" event={"ID":"b361f6202434cbe9c94eb392281c4180","Type":"ContainerStarted","Data":"4e664b99c4b984fb77cfc2dc76a56cc7d5eaa551702251b49c459bc185ae5c39"} Oct 31 21:16:00.078731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078682 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"48e3e5e26418af9437f262086f802d3b","Type":"ContainerStarted","Data":"b5ec9fe57064e5906d8c62e7fd300d56a68ddc58f125eccf0f2d2783c7f22ca5"} Oct 31 21:16:00.078731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078598 4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.078731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.078699 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" event={"ID":"48e3e5e26418af9437f262086f802d3b","Type":"ContainerStarted","Data":"a71d860626e52967c2938b14587329d1d71e3c0f7c1a61e72055a484ff058c8b"} Oct 31 21:16:00.101767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.101751 4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:16:00.101829 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:16:00.101786 4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.102416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.102402 4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:16:00.102451 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:16:00.102432 4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.126536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.126517 4294 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 21:16:00.127929 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.127914 4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientMemory" Oct 31 21:16:00.127985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.127937 4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 21:16:00.127985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.127948 4294 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-156.ec2.internal" event="NodeHasSufficientPID" Oct 31 21:16:00.127985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.127976 4294 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.164748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.164732 4294 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.164792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.164775 4294 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.196814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.196792 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.196879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.196816 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.196879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.196838 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48e3e5e26418af9437f262086f802d3b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"48e3e5e26418af9437f262086f802d3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.297067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.297042 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.297156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.297105 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.297156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.297127 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.297156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.297153 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48e3e5e26418af9437f262086f802d3b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"48e3e5e26418af9437f262086f802d3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.297246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.297181 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/48e3e5e26418af9437f262086f802d3b-config\") pod \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" (UID: \"48e3e5e26418af9437f262086f802d3b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.297246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.297218 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b361f6202434cbe9c94eb392281c4180-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" (UID: \"b361f6202434cbe9c94eb392281c4180\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:00.933215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.933178 4294 apiserver.go:52] "Watching apiserver" Oct 31 21:16:00.960743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.960713 4294 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 21:16:00.961098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.961080 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal","openshift-cluster-node-tuning-operator/tuned-jg7jt","openshift-image-registry/node-ca-2v76n","openshift-multus/network-metrics-daemon-mln5r","kube-system/global-pull-secret-syncer-29psw","openshift-dns/node-resolver-74gq6","openshift-network-diagnostics/network-check-target-z2f89","openshift-network-operator/iptables-alerter-9q252","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal","openshift-multus/multus-additional-cni-plugins-xdwqq","openshift-multus/multus-vbk79","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd","openshift-ovn-kubernetes/ovnkube-node-zd6fn","kube-system/konnectivity-agent-qwzj8"] Oct 31 21:16:00.962243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.962221 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:16:00.976668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976625 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.976668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976628 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.976668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976620 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 21:16:00.976668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976666 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 21:16:00.976951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976696 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.976951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976911 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.976951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976943 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 21:16:00.977095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.976955 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.977095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.977009 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.977095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.977037 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 21:16:00.977095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.977075 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.987012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.986992 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.987676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.987652 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 21:16:00.987782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.987762 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.987869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.987853 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 21:16:00.988115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988098 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.988185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988160 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.988185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.987767 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 21:16:00.988272 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988260 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.988330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988319 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.988402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988385 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.988457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988449 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 21:16:00.988499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988386 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:16:00.988499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988494 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 21:16:00.988586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988541 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 21:16:00.988778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.988760 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 21:16:00.991686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991667 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 21:16:00.991686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991676 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sflcb\"" Oct 31 21:16:00.991815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991788 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j68t2\"" Oct 31 21:16:00.991815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991791 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 21:16:00.991921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991903 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 21:16:00.991982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991920 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vddbx\"" Oct 31 21:16:00.991982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991941 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-rdjnn\"" Oct 31 21:16:00.991982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991965 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xdqcm\"" Oct 31 21:16:00.991982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991970 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9gmld\"" Oct 31 21:16:00.992139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991991 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6mmhq\"" Oct 31 21:16:00.992139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991996 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p85m4\"" Oct 31 21:16:00.992139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.992006 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvv8l\"" Oct 31 21:16:00.992139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.991965 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 21:16:00.992139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.992074 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 21:16:00.992139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.992088 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ddmwn\"" Oct 31 21:16:00.992317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.992147 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xsvd6\"" Oct 31 21:16:00.992317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:00.992172 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g9h45\"" Oct 31 21:16:01.011374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.011352 4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:01.011484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.011470 4294 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:16:01.034771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.034748 4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:16:01.034870 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:16:01.034812 4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-156.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-156.ec2.internal" Oct 31 21:16:01.035678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.035662 4294 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 21:16:01.035744 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:16:01.035698 4294 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-156.ec2.internal" Oct 31 21:16:01.045524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.045500 4294 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 21:16:01.100464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100434 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-socket-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.100571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100467 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-etc-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.100571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100494 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.100571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100517 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-host\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.100571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100552 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.100772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100590 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-ovn\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.100772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100635 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-modprobe-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.100772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100669 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-systemd\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.100772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100701 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-run\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.100772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100728 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-bin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.100772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100752 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-kubelet-config\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.101027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100775 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-kubernetes\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.101027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100797 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.101027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100820 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-slash\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100843 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-bin\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100905 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-config\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100932 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cni-binary-copy\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.101027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.100958 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-var-lib-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101048 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fea989-b58f-42e5-b17a-a0d35ab2161a-iptables-alerter-script\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101078 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-conf\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101094 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/751552d9-639f-42f2-bdcb-06fd195d7c2e-agent-certs\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101135 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-config\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101154 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101180 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-multus-certs\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101197 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101219 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-host\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101256 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-socket-dir-parent\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101270 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cni-binary-copy\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.101333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101284 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-systemd-units\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101355 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-log-socket\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101397 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-sys\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101423 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-lib-modules\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101433 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/751552d9-639f-42f2-bdcb-06fd195d7c2e-agent-certs\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101446 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-var-lib-kubelet\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101477 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101493 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101516 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/28fea989-b58f-42e5-b17a-a0d35ab2161a-iptables-alerter-script\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101502 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cnibin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101568 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-k8s-cni-cncf-io\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101618 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-env-overrides\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101628 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-binary-copy\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101653 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101688 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fea989-b58f-42e5-b17a-a0d35ab2161a-host-slash\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101712 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mx2\" (UniqueName: \"kubernetes.io/projected/28fea989-b58f-42e5-b17a-a0d35ab2161a-kube-api-access-74mx2\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:16:01.101730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101735 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-env-overrides\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101733 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-tmp\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101773 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzc8n\" (UniqueName: \"kubernetes.io/projected/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-kube-api-access-nzc8n\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101801 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfa483f-5315-4500-8f1b-e42d50e9b31b-hosts-file\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101821 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101869 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-tmp\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101880 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-original-pull-secret\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101965 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqxwz\" (UniqueName: \"kubernetes.io/projected/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-kube-api-access-pqxwz\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101977 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.101994 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-system-cni-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102012 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-netns\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102028 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-kubelet\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102060 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-hostroot\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f8jh\" (UniqueName: \"kubernetes.io/projected/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-kube-api-access-6f8jh\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102117 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-device-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102181 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-sys-fs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102217 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-kubelet\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102235 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-dbus\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102261 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-tuned\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102281 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-registration-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102330 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-systemd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102333 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-tuned\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.102392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102376 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102411 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovn-node-metrics-cert\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102444 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bvs\" (UniqueName: \"kubernetes.io/projected/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-kube-api-access-75bvs\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102489 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cnibin\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102576 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-conf-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102611 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-netns\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102650 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-node-log\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102679 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102712 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfmj\" (UniqueName: \"kubernetes.io/projected/cfc3bafa-8448-4f87-8480-f7311a31d1b4-kube-api-access-wvfmj\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102730 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovn-node-metrics-cert\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102736 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-system-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102757 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-multus\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102787 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-etc-kubernetes\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102812 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-serviceca\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102941 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.102971 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103012 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-daemon-config\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.103068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103037 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbzgs\" (UniqueName: \"kubernetes.io/projected/311ad56a-c14b-43a5-8ada-505326087f8f-kube-api-access-nbzgs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103116 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/751552d9-639f-42f2-bdcb-06fd195d7c2e-konnectivity-ca\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103149 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysconfig\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103183 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfa483f-5315-4500-8f1b-e42d50e9b31b-tmp-dir\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103211 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103234 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-script-lib\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103262 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-daemon-config\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103266 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfr4n\" (UniqueName: \"kubernetes.io/projected/76b1019f-cd75-49a1-bc25-60e0788a8f73-kube-api-access-gfr4n\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103326 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxnsf\" (UniqueName: \"kubernetes.io/projected/bbfa483f-5315-4500-8f1b-e42d50e9b31b-kube-api-access-kxnsf\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103332 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/751552d9-639f-42f2-bdcb-06fd195d7c2e-konnectivity-ca\") pod \"konnectivity-agent-qwzj8\" (UID: \"751552d9-639f-42f2-bdcb-06fd195d7c2e\") " pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103353 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-netd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103380 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103409 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfa483f-5315-4500-8f1b-e42d50e9b31b-tmp-dir\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103424 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-serviceca\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:16:01.103702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.103459 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cfc3bafa-8448-4f87-8480-f7311a31d1b4-ovnkube-script-lib\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.122551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.122530 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74mx2\" (UniqueName: \"kubernetes.io/projected/28fea989-b58f-42e5-b17a-a0d35ab2161a-kube-api-access-74mx2\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:16:01.122647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.122631 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzc8n\" (UniqueName: \"kubernetes.io/projected/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-kube-api-access-nzc8n\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.122726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.122710 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqxwz\" (UniqueName: \"kubernetes.io/projected/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-kube-api-access-pqxwz\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.123389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.123357 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f8jh\" (UniqueName: \"kubernetes.io/projected/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-kube-api-access-6f8jh\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.123597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.123577 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbzgs\" (UniqueName: \"kubernetes.io/projected/311ad56a-c14b-43a5-8ada-505326087f8f-kube-api-access-nbzgs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.123681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.123664 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bvs\" (UniqueName: \"kubernetes.io/projected/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-kube-api-access-75bvs\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:16:01.123818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.123802 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfmj\" (UniqueName: \"kubernetes.io/projected/cfc3bafa-8448-4f87-8480-f7311a31d1b4-kube-api-access-wvfmj\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.124891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.124869 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfr4n\" (UniqueName: \"kubernetes.io/projected/76b1019f-cd75-49a1-bc25-60e0788a8f73-kube-api-access-gfr4n\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:16:01.125123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.125107 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m85cj\" (UniqueName: \"kubernetes.io/projected/8243464a-58b2-4d49-8004-72ad5bd9c085-kube-api-access-m85cj\") pod \"network-check-target-z2f89\" (UID: \"8243464a-58b2-4d49-8004-72ad5bd9c085\") " pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:16:01.125166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.125137 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxnsf\" (UniqueName: \"kubernetes.io/projected/bbfa483f-5315-4500-8f1b-e42d50e9b31b-kube-api-access-kxnsf\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:16:01.203800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203748 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.203800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203774 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-socket-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.203800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203789 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-etc-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203803 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203834 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-host\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203860 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-socket-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203876 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-host\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203881 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-etc-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203897 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203898 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203915 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-ovn\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203938 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-ovn\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203941 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203946 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-modprobe-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203968 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203972 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-systemd\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.203997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.203994 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-run\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204017 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-bin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204035 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-systemd\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204042 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-kubelet-config\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204049 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-modprobe-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204054 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-run\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204068 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-bin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204066 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-kubernetes\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204097 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-kubelet-config\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204113 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-kubernetes\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204115 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-slash\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204138 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-slash\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204149 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-bin\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204175 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-var-lib-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204193 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-d\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204185 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-bin\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204202 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-conf\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204230 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-var-lib-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204229 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-multus-certs\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204256 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204268 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-multus-certs\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204271 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-host\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204294 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4af55054-ae66-45a2-a8ea-4a22b5e95bfe-host\") pod \"node-ca-2v76n\" (UID: \"4af55054-ae66-45a2-a8ea-4a22b5e95bfe\") " pod="openshift-image-registry/node-ca-2v76n" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204319 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-socket-dir-parent\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204327 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-etc-selinux\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204345 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-systemd-units\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204360 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-socket-dir-parent\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204368 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-log-socket\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204393 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-sys\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204399 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-systemd-units\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204412 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysctl-conf\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204415 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-lib-modules\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204433 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-log-socket\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204440 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-var-lib-kubelet\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204446 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-sys\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204458 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cnibin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204480 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-k8s-cni-cncf-io\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204508 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fea989-b58f-42e5-b17a-a0d35ab2161a-host-slash\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204513 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-var-lib-kubelet\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204520 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-lib-modules\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204531 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-cnibin\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204534 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfa483f-5315-4500-8f1b-e42d50e9b31b-hosts-file\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204553 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28fea989-b58f-42e5-b17a-a0d35ab2161a-host-slash\") pod \"iptables-alerter-9q252\" (UID: \"28fea989-b58f-42e5-b17a-a0d35ab2161a\") " pod="openshift-network-operator/iptables-alerter-9q252" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204568 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-system-cni-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204571 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfa483f-5315-4500-8f1b-e42d50e9b31b-hosts-file\") pod \"node-resolver-74gq6\" (UID: \"bbfa483f-5315-4500-8f1b-e42d50e9b31b\") " pod="openshift-dns/node-resolver-74gq6" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204593 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-netns\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204598 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-system-cni-dir\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204540 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-k8s-cni-cncf-io\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.204624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204632 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-run-netns\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204702 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-kubelet\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.204997 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-kubelet\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205021 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-hostroot\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205037 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-device-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205051 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-sys-fs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205064 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-kubelet\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205077 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-dbus\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205092 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-registration-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205107 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-systemd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205116 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-kubelet\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205119 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-device-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205135 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-hostroot\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205122 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205166 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bc8d0364-096a-49f7-87b3-ae72ae71f1a1-dbus\") pod \"global-pull-secret-syncer-29psw\" (UID: \"bc8d0364-096a-49f7-87b3-ae72ae71f1a1\") " pod="kube-system/global-pull-secret-syncer-29psw" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205173 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-openvswitch\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205178 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-sys-fs\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205180 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cnibin\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205176 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/311ad56a-c14b-43a5-8ada-505326087f8f-registration-dir\") pod \"aws-ebs-csi-driver-node-4slwd\" (UID: \"311ad56a-c14b-43a5-8ada-505326087f8f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4slwd" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205202 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-run-systemd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205208 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-conf-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205232 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-netns\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205257 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d778febb-a3d7-4ddc-a5de-cdcefe1e3952-cnibin\") pod \"multus-additional-cni-plugins-xdwqq\" (UID: \"d778febb-a3d7-4ddc-a5de-cdcefe1e3952\") " pod="openshift-multus/multus-additional-cni-plugins-xdwqq" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205260 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-run-netns\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205268 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-conf-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205277 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-node-log\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205295 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205334 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-system-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205350 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-multus\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205352 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-node-log\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205381 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-host-var-lib-cni-multus\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205380 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205391 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-system-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205410 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-etc-kubernetes\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205424 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205438 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205443 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-etc-kubernetes\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205454 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysconfig\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205476 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-netd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205498 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d5d7007d-ab88-47fb-a484-b6ee82e1f5ff-multus-cni-dir\") pod \"multus-vbk79\" (UID: \"d5d7007d-ab88-47fb-a484-b6ee82e1f5ff\") " pod="openshift-multus/multus-vbk79" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205510 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cfc3bafa-8448-4f87-8480-f7311a31d1b4-host-cni-netd\") pod \"ovnkube-node-zd6fn\" (UID: \"cfc3bafa-8448-4f87-8480-f7311a31d1b4\") " pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205518 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9a70fca7-fa0b-4b05-abbb-cb6258e84ae0-etc-sysconfig\") pod \"tuned-jg7jt\" (UID: \"9a70fca7-fa0b-4b05-abbb-cb6258e84ae0\") " pod="openshift-cluster-node-tuning-operator/tuned-jg7jt" Oct 31 21:16:01.206122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.205722 4294 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 21:16:01.207650 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.207635 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76b1019f-cd75-49a1-bc25-60e0788a8f73-metrics-certs\") pod \"network-metrics-daemon-mln5r\" (UID: \"76b1019f-cd75-49a1-bc25-60e0788a8f73\") " pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:16:01.262885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.262864 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mln5r" Oct 31 21:16:01.581563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.581538 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mln5r"] Oct 31 21:16:01.587686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:01.587635 4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:16:02.014096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:02.014061 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mln5r" event={"ID":"76b1019f-cd75-49a1-bc25-60e0788a8f73","Type":"ContainerStarted","Data":"65827142aad76eb8a1f6ddb8ec0f610a37bfaf1165144491b1215256e71ee140"} Oct 31 21:16:03.017644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:03.017601 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mln5r" event={"ID":"76b1019f-cd75-49a1-bc25-60e0788a8f73","Type":"ContainerStarted","Data":"dbaabf4e687d9ec4b7021951d8ad40951c455fa493a5f3cbf630c62873e8dc4b"} Oct 31 21:16:03.018018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:03.017665 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mln5r" event={"ID":"76b1019f-cd75-49a1-bc25-60e0788a8f73","Type":"ContainerStarted","Data":"fbfa93a8438a92b6870b796999431522b6a0fc08db58db23800779da5f0a2177"} Oct 31 21:16:06.131394 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:06.131348 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:16:07.209895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:07.209866 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:07.707640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:07.707607 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:08.030127 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:08.030102 4294 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 21:16:09.718057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.718017 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k6sw4"] Oct 31 21:16:09.719837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.719817 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h"] Oct 31 21:16:09.719962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.719941 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.721248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.721232 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" Oct 31 21:16:09.731171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.731141 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-26zf5"] Oct 31 21:16:09.733026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.733007 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:09.740874 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.739891 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h"] Oct 31 21:16:09.740874 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.740359 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6sw4"] Oct 31 21:16:09.744680 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.744655 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 21:16:09.753069 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.753043 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 21:16:09.753681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.753665 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 21:16:09.754960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.754931 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-69h6c\"" Oct 31 21:16:09.754960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.754935 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dwlts\"" Oct 31 21:16:09.755284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.755266 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 21:16:09.755369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.755294 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 21:16:09.755834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.755819 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 21:16:09.756870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.756853 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-nq8jq\"" Oct 31 21:16:09.765806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.765774 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vx47h"] Oct 31 21:16:09.767823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.767806 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26zf5"] Oct 31 21:16:09.767918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.767907 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.788471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.788437 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vx47h"] Oct 31 21:16:09.789973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.789932 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 21:16:09.790093 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.790032 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 21:16:09.790093 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.790061 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 21:16:09.835754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.835719 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c59468d77-4655d"] Oct 31 21:16:09.837691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.837673 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.849021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.848991 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxstl\" (UniqueName: \"kubernetes.io/projected/f2e9e4d9-96d7-4c18-a37f-be880f5b511e-kube-api-access-dxstl\") pod \"ingress-canary-26zf5\" (UID: \"f2e9e4d9-96d7-4c18-a37f-be880f5b511e\") " pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:09.849159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849027 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72e88409-4f64-46c1-8818-d625e4411683-config-volume\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.849159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849049 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7147b14-f465-4a20-8618-708bad6bb6db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.849159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849103 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7147b14-f465-4a20-8618-708bad6bb6db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.849159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849151 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cp8m\" (UniqueName: \"kubernetes.io/projected/d7147b14-f465-4a20-8618-708bad6bb6db-kube-api-access-9cp8m\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.849286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849201 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmgz\" (UniqueName: \"kubernetes.io/projected/72e88409-4f64-46c1-8818-d625e4411683-kube-api-access-ffmgz\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.849286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849233 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e88409-4f64-46c1-8818-d625e4411683-tmp-dir\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.849286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849254 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72e88409-4f64-46c1-8818-d625e4411683-metrics-tls\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.849404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849314 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e9e4d9-96d7-4c18-a37f-be880f5b511e-cert\") pod \"ingress-canary-26zf5\" (UID: \"f2e9e4d9-96d7-4c18-a37f-be880f5b511e\") " pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:09.849404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849382 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ed8fd589-3f8a-4ede-9a1b-2749f7a98395-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f945h\" (UID: \"ed8fd589-3f8a-4ede-9a1b-2749f7a98395\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" Oct 31 21:16:09.849465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849404 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7147b14-f465-4a20-8618-708bad6bb6db-crio-socket\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.849465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.849419 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7147b14-f465-4a20-8618-708bad6bb6db-data-volume\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.865181 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.865143 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 21:16:09.865402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.865222 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 21:16:09.865402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.865274 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-866d2\"" Oct 31 21:16:09.865402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.865334 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 21:16:09.865580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.865436 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-28gfg\"" Oct 31 21:16:09.865580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.865534 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 21:16:09.865746 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.865721 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 21:16:09.866048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.866028 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c59468d77-4655d"] Oct 31 21:16:09.932111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.932077 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qwzj8" Oct 31 21:16:09.949864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.949827 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e88409-4f64-46c1-8818-d625e4411683-tmp-dir\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.949864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.949867 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7418585-f650-4670-87ba-241b6752a329-trusted-ca\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.950116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.949890 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72e88409-4f64-46c1-8818-d625e4411683-metrics-tls\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.950116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.949914 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7418585-f650-4670-87ba-241b6752a329-registry-certificates\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.950116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.949970 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7418585-f650-4670-87ba-241b6752a329-installation-pull-secrets\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.950116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950081 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e9e4d9-96d7-4c18-a37f-be880f5b511e-cert\") pod \"ingress-canary-26zf5\" (UID: \"f2e9e4d9-96d7-4c18-a37f-be880f5b511e\") " pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:09.950344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950151 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ed8fd589-3f8a-4ede-9a1b-2749f7a98395-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f945h\" (UID: \"ed8fd589-3f8a-4ede-9a1b-2749f7a98395\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" Oct 31 21:16:09.950344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950181 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7147b14-f465-4a20-8618-708bad6bb6db-crio-socket\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.950344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950206 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7147b14-f465-4a20-8618-708bad6bb6db-data-volume\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.950344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950231 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-bound-sa-token\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.950344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950232 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72e88409-4f64-46c1-8818-d625e4411683-tmp-dir\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.950344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950263 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btmxj\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-kube-api-access-btmxj\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.950344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950321 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7418585-f650-4670-87ba-241b6752a329-image-registry-private-configuration\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950324 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7147b14-f465-4a20-8618-708bad6bb6db-crio-socket\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950463 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxstl\" (UniqueName: \"kubernetes.io/projected/f2e9e4d9-96d7-4c18-a37f-be880f5b511e-kube-api-access-dxstl\") pod \"ingress-canary-26zf5\" (UID: \"f2e9e4d9-96d7-4c18-a37f-be880f5b511e\") " pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950500 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72e88409-4f64-46c1-8818-d625e4411683-config-volume\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950520 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7147b14-f465-4a20-8618-708bad6bb6db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950540 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7147b14-f465-4a20-8618-708bad6bb6db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950559 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7147b14-f465-4a20-8618-708bad6bb6db-data-volume\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950569 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cp8m\" (UniqueName: \"kubernetes.io/projected/d7147b14-f465-4a20-8618-708bad6bb6db-kube-api-access-9cp8m\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950594 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmgz\" (UniqueName: \"kubernetes.io/projected/72e88409-4f64-46c1-8818-d625e4411683-kube-api-access-ffmgz\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950620 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-registry-tls\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.950723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.950645 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7418585-f650-4670-87ba-241b6752a329-ca-trust-extracted\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:09.951497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.951469 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/72e88409-4f64-46c1-8818-d625e4411683-config-volume\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.951636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.951601 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7147b14-f465-4a20-8618-708bad6bb6db-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.952533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.952508 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/72e88409-4f64-46c1-8818-d625e4411683-metrics-tls\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.952739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.952721 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ed8fd589-3f8a-4ede-9a1b-2749f7a98395-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f945h\" (UID: \"ed8fd589-3f8a-4ede-9a1b-2749f7a98395\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" Oct 31 21:16:09.952782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.952740 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2e9e4d9-96d7-4c18-a37f-be880f5b511e-cert\") pod \"ingress-canary-26zf5\" (UID: \"f2e9e4d9-96d7-4c18-a37f-be880f5b511e\") " pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:09.953071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.953055 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7147b14-f465-4a20-8618-708bad6bb6db-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.974260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.974190 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cp8m\" (UniqueName: \"kubernetes.io/projected/d7147b14-f465-4a20-8618-708bad6bb6db-kube-api-access-9cp8m\") pod \"insights-runtime-extractor-vx47h\" (UID: \"d7147b14-f465-4a20-8618-708bad6bb6db\") " pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:09.974410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.974270 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmgz\" (UniqueName: \"kubernetes.io/projected/72e88409-4f64-46c1-8818-d625e4411683-kube-api-access-ffmgz\") pod \"dns-default-k6sw4\" (UID: \"72e88409-4f64-46c1-8818-d625e4411683\") " pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:09.974410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:09.974330 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxstl\" (UniqueName: \"kubernetes.io/projected/f2e9e4d9-96d7-4c18-a37f-be880f5b511e-kube-api-access-dxstl\") pod \"ingress-canary-26zf5\" (UID: \"f2e9e4d9-96d7-4c18-a37f-be880f5b511e\") " pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:10.031419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.031380 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:10.037727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.037703 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" Oct 31 21:16:10.044806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.044782 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-26zf5" Oct 31 21:16:10.050958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.050925 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-bound-sa-token\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.050989 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btmxj\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-kube-api-access-btmxj\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.051030 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7418585-f650-4670-87ba-241b6752a329-image-registry-private-configuration\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.051093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-registry-tls\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.051130 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7418585-f650-4670-87ba-241b6752a329-ca-trust-extracted\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.051174 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7418585-f650-4670-87ba-241b6752a329-trusted-ca\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.051204 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7418585-f650-4670-87ba-241b6752a329-registry-certificates\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.051230 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7418585-f650-4670-87ba-241b6752a329-installation-pull-secrets\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.051828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.051761 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c7418585-f650-4670-87ba-241b6752a329-ca-trust-extracted\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.053339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.053295 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c7418585-f650-4670-87ba-241b6752a329-registry-certificates\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.053339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.053331 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c7418585-f650-4670-87ba-241b6752a329-trusted-ca\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.054632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.054609 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c7418585-f650-4670-87ba-241b6752a329-image-registry-private-configuration\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.054898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.054873 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c7418585-f650-4670-87ba-241b6752a329-installation-pull-secrets\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.056201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.056172 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-registry-tls\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.077845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.077804 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btmxj\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-kube-api-access-btmxj\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.079184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.079148 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c7418585-f650-4670-87ba-241b6752a329-bound-sa-token\") pod \"image-registry-c59468d77-4655d\" (UID: \"c7418585-f650-4670-87ba-241b6752a329\") " pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.079369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.079315 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vx47h" Oct 31 21:16:10.147796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.147706 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:10.150421 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.149853 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zd6fn" Oct 31 21:16:10.343050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.343017 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6sw4"] Oct 31 21:16:10.345799 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:10.345766 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e88409_4f64_46c1_8818_d625e4411683.slice/crio-fe6ff6ffe19f2a93e1828cd5407deb21e7c686d67106cf9b0ab39db7528b6bb4 WatchSource:0}: Error finding container fe6ff6ffe19f2a93e1828cd5407deb21e7c686d67106cf9b0ab39db7528b6bb4: Status 404 returned error can't find the container with id fe6ff6ffe19f2a93e1828cd5407deb21e7c686d67106cf9b0ab39db7528b6bb4 Oct 31 21:16:10.368929 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.368904 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h"] Oct 31 21:16:10.371142 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:10.371109 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8fd589_3f8a_4ede_9a1b_2749f7a98395.slice/crio-3abcce660943b86890775eabd07bb955bfc3607d5daf99b11b02bc5a303c81fa WatchSource:0}: Error finding container 3abcce660943b86890775eabd07bb955bfc3607d5daf99b11b02bc5a303c81fa: Status 404 returned error can't find the container with id 3abcce660943b86890775eabd07bb955bfc3607d5daf99b11b02bc5a303c81fa Oct 31 21:16:10.378872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.378846 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-26zf5"] Oct 31 21:16:10.381255 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:10.381220 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e9e4d9_96d7_4c18_a37f_be880f5b511e.slice/crio-575e62bb02b902b4c167e563219c663c2f5178fe02aa0f830cca626beac09467 WatchSource:0}: Error finding container 575e62bb02b902b4c167e563219c663c2f5178fe02aa0f830cca626beac09467: Status 404 returned error can't find the container with id 575e62bb02b902b4c167e563219c663c2f5178fe02aa0f830cca626beac09467 Oct 31 21:16:10.441684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.441661 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vx47h"] Oct 31 21:16:10.443990 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:10.443965 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7147b14_f465_4a20_8618_708bad6bb6db.slice/crio-ea3fa3f6f987e9fbecab255edea323d7fe55e9b7f0ff65c0c4464f74b102c1b5 WatchSource:0}: Error finding container ea3fa3f6f987e9fbecab255edea323d7fe55e9b7f0ff65c0c4464f74b102c1b5: Status 404 returned error can't find the container with id ea3fa3f6f987e9fbecab255edea323d7fe55e9b7f0ff65c0c4464f74b102c1b5 Oct 31 21:16:10.464804 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:10.464781 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c59468d77-4655d"] Oct 31 21:16:10.476555 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:10.476526 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7418585_f650_4670_87ba_241b6752a329.slice/crio-bb4f9673c5776146da0a78e342638a83000bd0b47d8d3870caaa1def80b5e539 WatchSource:0}: Error finding container bb4f9673c5776146da0a78e342638a83000bd0b47d8d3870caaa1def80b5e539: Status 404 returned error can't find the container with id bb4f9673c5776146da0a78e342638a83000bd0b47d8d3870caaa1def80b5e539 Oct 31 21:16:11.045773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.045681 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" event={"ID":"ed8fd589-3f8a-4ede-9a1b-2749f7a98395","Type":"ContainerStarted","Data":"3abcce660943b86890775eabd07bb955bfc3607d5daf99b11b02bc5a303c81fa"} Oct 31 21:16:11.049722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.049684 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vx47h" event={"ID":"d7147b14-f465-4a20-8618-708bad6bb6db","Type":"ContainerStarted","Data":"b7529d6b69ce9f21515986d4a9f71b6cde2db90cd2d83a64105609c0528389a6"} Oct 31 21:16:11.049895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.049762 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vx47h" event={"ID":"d7147b14-f465-4a20-8618-708bad6bb6db","Type":"ContainerStarted","Data":"ea3fa3f6f987e9fbecab255edea323d7fe55e9b7f0ff65c0c4464f74b102c1b5"} Oct 31 21:16:11.051917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.051881 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6sw4" event={"ID":"72e88409-4f64-46c1-8818-d625e4411683","Type":"ContainerStarted","Data":"fe6ff6ffe19f2a93e1828cd5407deb21e7c686d67106cf9b0ab39db7528b6bb4"} Oct 31 21:16:11.055129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.055089 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c59468d77-4655d" event={"ID":"c7418585-f650-4670-87ba-241b6752a329","Type":"ContainerStarted","Data":"afb2b007ff7ab543117d08ea954b0e893d86b0ad26ce87184bb878237a8d329d"} Oct 31 21:16:11.055268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.055148 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c59468d77-4655d" event={"ID":"c7418585-f650-4670-87ba-241b6752a329","Type":"ContainerStarted","Data":"bb4f9673c5776146da0a78e342638a83000bd0b47d8d3870caaa1def80b5e539"} Oct 31 21:16:11.055268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.055214 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:11.057101 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.057072 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26zf5" event={"ID":"f2e9e4d9-96d7-4c18-a37f-be880f5b511e","Type":"ContainerStarted","Data":"575e62bb02b902b4c167e563219c663c2f5178fe02aa0f830cca626beac09467"} Oct 31 21:16:11.102252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:11.102194 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c59468d77-4655d" podStartSLOduration=2.102174986 podStartE2EDuration="2.102174986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:11.101438717 +0000 UTC m=+11.226545949" watchObservedRunningTime="2026-10-31 21:16:11.102174986 +0000 UTC m=+11.227282217" Oct 31 21:16:12.078194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:12.078101 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vx47h" event={"ID":"d7147b14-f465-4a20-8618-708bad6bb6db","Type":"ContainerStarted","Data":"061feacf974fdb302fdca2ecb186f8c42611871a026f38170cabe0d47b54e95c"} Oct 31 21:16:12.080473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:12.080437 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" event={"ID":"ed8fd589-3f8a-4ede-9a1b-2749f7a98395","Type":"ContainerStarted","Data":"6f3f1207d309a728ffc5319e4827111c3c6fa65c62ff48765bdb5f68665da839"} Oct 31 21:16:12.121827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:12.121762 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" podStartSLOduration=1.739205922 podStartE2EDuration="3.121745724s" totalImagesPullingTime="1.382539802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:12.120998999 +0000 UTC m=+12.246106236" watchObservedRunningTime="2026-10-31 21:16:12.121745724 +0000 UTC m=+12.246852953" Oct 31 21:16:13.086324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.086275 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vx47h" event={"ID":"d7147b14-f465-4a20-8618-708bad6bb6db","Type":"ContainerStarted","Data":"d451d3c7e0b6b21b8370672e754f94d40f904cd1458ee621313f7ea0861ccf87"} Oct 31 21:16:13.088789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.088754 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6sw4" event={"ID":"72e88409-4f64-46c1-8818-d625e4411683","Type":"ContainerStarted","Data":"31513614ad360635120fc52c9c5777c663abba14a2dd419fb60b80c4a5e5d58f"} Oct 31 21:16:13.088934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.088817 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6sw4" event={"ID":"72e88409-4f64-46c1-8818-d625e4411683","Type":"ContainerStarted","Data":"9641d33ce3e2176f8d91abbd5a366bf2bea96c75ed27cb182b4b22295dc0d4b7"} Oct 31 21:16:13.089045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.089030 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:13.089084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.089070 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" Oct 31 21:16:13.094562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.094534 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f945h" Oct 31 21:16:13.130070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.129976 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vx47h" podStartSLOduration=2.059274612 podStartE2EDuration="4.129956604s" totalImagesPullingTime="2.070681992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 21:16:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:13.129712503 +0000 UTC m=+13.254819735" watchObservedRunningTime="2026-10-31 21:16:13.129956604 +0000 UTC m=+13.255063848" Oct 31 21:16:13.171654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:13.171589 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-k6sw4" podStartSLOduration=2.077929943 podStartE2EDuration="4.171571769s" totalImagesPullingTime="2.093641826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:13.171260086 +0000 UTC m=+13.296367330" watchObservedRunningTime="2026-10-31 21:16:13.171571769 +0000 UTC m=+13.296679005" Oct 31 21:16:14.093783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:14.093740 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-26zf5" event={"ID":"f2e9e4d9-96d7-4c18-a37f-be880f5b511e","Type":"ContainerStarted","Data":"e83351c307c10f13c44bc4c92abeaadf288c15524617a7b9c1688d55bbcb76f0"} Oct 31 21:16:14.134620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:14.134567 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-26zf5" podStartSLOduration=2.300500171 podStartE2EDuration="5.134550535s" totalImagesPullingTime="2.834050364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:14.13387814 +0000 UTC m=+14.258985365" watchObservedRunningTime="2026-10-31 21:16:14.134550535 +0000 UTC m=+14.259657766" Oct 31 21:16:18.595829 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.595791 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-plnrb"] Oct 31 21:16:18.598832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.598810 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.636529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.636494 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 21:16:18.636726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.636662 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 21:16:18.636726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.636693 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xn5d6\"" Oct 31 21:16:18.639318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.639285 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 21:16:18.639413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.639327 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 21:16:18.639413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.639389 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 21:16:18.648612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.648588 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 21:16:18.724904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.724860 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-textfile\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.724904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.724906 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-wtmp\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.725109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.724999 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-accelerators-collector-config\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.725109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.725063 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19000f2f-b5f2-4213-9f97-3928d2880662-metrics-client-ca\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.725219 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.725096 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkclw\" (UniqueName: \"kubernetes.io/projected/19000f2f-b5f2-4213-9f97-3928d2880662-kube-api-access-pkclw\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.725219 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.725194 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-tls\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.725284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.725249 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.725284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.725277 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-sys\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.725393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.725329 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-root\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826607 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19000f2f-b5f2-4213-9f97-3928d2880662-metrics-client-ca\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826659 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkclw\" (UniqueName: \"kubernetes.io/projected/19000f2f-b5f2-4213-9f97-3928d2880662-kube-api-access-pkclw\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826680 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-tls\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826699 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826719 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-sys\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826777 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-sys\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826833 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-root\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826867 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-textfile\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.826907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826891 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-wtmp\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.827238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826918 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-accelerators-collector-config\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.827238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.826948 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-root\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.827238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.827051 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-wtmp\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.827357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.827267 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-textfile\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.827446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.827428 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19000f2f-b5f2-4213-9f97-3928d2880662-metrics-client-ca\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.827567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.827547 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-accelerators-collector-config\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.829062 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.829044 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.829268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.829249 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19000f2f-b5f2-4213-9f97-3928d2880662-node-exporter-tls\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.849620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.849557 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkclw\" (UniqueName: \"kubernetes.io/projected/19000f2f-b5f2-4213-9f97-3928d2880662-kube-api-access-pkclw\") pod \"node-exporter-plnrb\" (UID: \"19000f2f-b5f2-4213-9f97-3928d2880662\") " pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.910910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:18.910875 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-plnrb" Oct 31 21:16:18.921044 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:18.921012 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19000f2f_b5f2_4213_9f97_3928d2880662.slice/crio-77021fca1b105e787fb731e4f9c0f8e95f7ff69fa53f1f8ab4dd879a1a52fabc WatchSource:0}: Error finding container 77021fca1b105e787fb731e4f9c0f8e95f7ff69fa53f1f8ab4dd879a1a52fabc: Status 404 returned error can't find the container with id 77021fca1b105e787fb731e4f9c0f8e95f7ff69fa53f1f8ab4dd879a1a52fabc Oct 31 21:16:19.112769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:19.112685 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-plnrb" event={"ID":"19000f2f-b5f2-4213-9f97-3928d2880662","Type":"ContainerStarted","Data":"77021fca1b105e787fb731e4f9c0f8e95f7ff69fa53f1f8ab4dd879a1a52fabc"} Oct 31 21:16:19.998454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:19.998421 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 21:16:20.000665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.000644 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.016175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.016148 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 21:16:20.016343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.016160 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 21:16:20.016450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.016160 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 21:16:20.028478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.028415 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 21:16:20.028478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.028426 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 21:16:20.028478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.028443 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 21:16:20.028478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.028465 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 21:16:20.028478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.028425 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 21:16:20.028794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.028609 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-c54qw\"" Oct 31 21:16:20.029507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.029484 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 21:16:20.041412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.041390 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 21:16:20.117413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.117378 4294 generic.go:356] "Generic (PLEG): container finished" podID="19000f2f-b5f2-4213-9f97-3928d2880662" containerID="51063ad8c81e71c8b93da228f3a1f931822af084a31286d40f329c98c2e2f66b" exitCode=0 Oct 31 21:16:20.117575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.117458 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-plnrb" event={"ID":"19000f2f-b5f2-4213-9f97-3928d2880662","Type":"ContainerDied","Data":"51063ad8c81e71c8b93da228f3a1f931822af084a31286d40f329c98c2e2f66b"} Oct 31 21:16:20.135183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135157 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135195 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-web-config\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135218 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135370 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135313 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135424 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135465 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w775\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-kube-api-access-6w775\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135519 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135600 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135559 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135690 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135669 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-out\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135706 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135732 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.135851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.135761 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237067 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237120 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-out\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237138 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237154 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237183 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237231 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237275 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-web-config\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237405 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237444 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237470 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237496 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237520 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w775\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-kube-api-access-6w775\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237552 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.237848 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.237785 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.239542 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.239514 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.240608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.240219 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.240719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.240616 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.240884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.240857 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-volume\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.240946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.240906 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.240946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.240914 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.241047 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.240984 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.241400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.241374 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-out\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.242468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.242444 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.242839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.242820 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.243164 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.243145 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-web-config\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.265776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.265745 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w775\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-kube-api-access-6w775\") pod \"alertmanager-main-0\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.317689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.317598 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:16:20.629081 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.629042 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 21:16:20.631660 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:20.631627 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008e71ad_8d93_4da1_81fc_0e9447810f1f.slice/crio-2babd457e0bee5bf88c86bbbc8267f0dbed82f6b5a81f32eeacd67d3163b55c3 WatchSource:0}: Error finding container 2babd457e0bee5bf88c86bbbc8267f0dbed82f6b5a81f32eeacd67d3163b55c3: Status 404 returned error can't find the container with id 2babd457e0bee5bf88c86bbbc8267f0dbed82f6b5a81f32eeacd67d3163b55c3 Oct 31 21:16:20.898999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.895948 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-555f8d6d4b-prhz2"] Oct 31 21:16:20.901161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.901129 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:20.919231 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.919190 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 21:16:20.919410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.919356 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 21:16:20.919410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.919356 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 21:16:20.919410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.919379 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 21:16:20.919410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.919365 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2evs4ghj4catt\"" Oct 31 21:16:20.919566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.919415 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 21:16:20.919566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.919493 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bsdp9\"" Oct 31 21:16:20.932860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:20.932824 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-555f8d6d4b-prhz2"] Oct 31 21:16:21.045032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.044991 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-grpc-tls\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045034 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045148 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b055db-6639-407e-a248-c1339b966438-metrics-client-ca\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045201 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045229 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26b055db-6639-407e-a248-c1339b966438-tmp\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045352 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045390 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjqrh\" (UniqueName: \"kubernetes.io/projected/26b055db-6639-407e-a248-c1339b966438-kube-api-access-jjqrh\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045450 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-tls\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.045544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.045472 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.123886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.123782 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-plnrb" event={"ID":"19000f2f-b5f2-4213-9f97-3928d2880662","Type":"ContainerStarted","Data":"43926e113f7585e7e65ea4ed207027be9b86bc88420bf70446c85a5fe7fa1273"} Oct 31 21:16:21.123886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.123848 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-plnrb" event={"ID":"19000f2f-b5f2-4213-9f97-3928d2880662","Type":"ContainerStarted","Data":"3934129ef903181bce9131c10f383376c3a55b2cfbd8ffb97ecfd68c813889f3"} Oct 31 21:16:21.125173 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.125143 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerStarted","Data":"2babd457e0bee5bf88c86bbbc8267f0dbed82f6b5a81f32eeacd67d3163b55c3"} Oct 31 21:16:21.146677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146636 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b055db-6639-407e-a248-c1339b966438-metrics-client-ca\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.146864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146694 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.146864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146725 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26b055db-6639-407e-a248-c1339b966438-tmp\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.146864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146777 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.146864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146804 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjqrh\" (UniqueName: \"kubernetes.io/projected/26b055db-6639-407e-a248-c1339b966438-kube-api-access-jjqrh\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.146864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146843 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-tls\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.147056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146869 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.147056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146902 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-grpc-tls\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.147056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.146929 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.147513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.147486 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/26b055db-6639-407e-a248-c1339b966438-metrics-client-ca\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.148123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.148095 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26b055db-6639-407e-a248-c1339b966438-tmp\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.150736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.150676 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.150827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.150792 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-tls\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.150898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.150859 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-grpc-tls\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.150957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.150926 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.151729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.151708 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.152026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.152007 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/26b055db-6639-407e-a248-c1339b966438-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.168156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.168095 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-plnrb" podStartSLOduration=2.255363965 podStartE2EDuration="3.168073026s" totalImagesPullingTime="845.54629ms" totalInitContainerRuntime="67.162771ms" isStatefulPod=true podCreationTimestamp="2026-10-31 21:16:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:21.167069216 +0000 UTC m=+21.292176450" watchObservedRunningTime="2026-10-31 21:16:21.168073026 +0000 UTC m=+21.293180259" Oct 31 21:16:21.170542 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.170510 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjqrh\" (UniqueName: \"kubernetes.io/projected/26b055db-6639-407e-a248-c1339b966438-kube-api-access-jjqrh\") pod \"thanos-querier-555f8d6d4b-prhz2\" (UID: \"26b055db-6639-407e-a248-c1339b966438\") " pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.217240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.217197 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:21.530605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:21.530580 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-555f8d6d4b-prhz2"] Oct 31 21:16:21.532100 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:21.532067 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b055db_6639_407e_a248_c1339b966438.slice/crio-d449b0febe5b81025cf982a2c011fd68f1707a4f7e9a67cd8f39c58ccc8ee0b4 WatchSource:0}: Error finding container d449b0febe5b81025cf982a2c011fd68f1707a4f7e9a67cd8f39c58ccc8ee0b4: Status 404 returned error can't find the container with id d449b0febe5b81025cf982a2c011fd68f1707a4f7e9a67cd8f39c58ccc8ee0b4 Oct 31 21:16:22.130796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:22.130756 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" event={"ID":"26b055db-6639-407e-a248-c1339b966438","Type":"ContainerStarted","Data":"d449b0febe5b81025cf982a2c011fd68f1707a4f7e9a67cd8f39c58ccc8ee0b4"} Oct 31 21:16:22.132655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:22.132621 4294 generic.go:356] "Generic (PLEG): container finished" podID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerID="93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8" exitCode=0 Oct 31 21:16:22.132788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:22.132707 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8"} Oct 31 21:16:23.096816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.096781 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k6sw4" Oct 31 21:16:23.239482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.239443 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8474bc4858-594xw"] Oct 31 21:16:23.241990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.241966 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.270573 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.270402 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 31 21:16:23.270573 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.270436 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 21:16:23.270573 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.270539 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 31 21:16:23.282273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.278515 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8474bc4858-594xw"] Oct 31 21:16:23.282273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.278797 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2d6dl\"" Oct 31 21:16:23.282273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.278894 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7rl79ehed05t0\"" Oct 31 21:16:23.282273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.278950 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 31 21:16:23.367946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.367854 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fac108a2-b5b6-4241-82a2-e180d756608a-audit-log\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.367946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.367920 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac108a2-b5b6-4241-82a2-e180d756608a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.368171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.368010 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-secret-metrics-server-client-certs\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.368171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.368079 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-client-ca-bundle\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.368171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.368156 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wzr\" (UniqueName: \"kubernetes.io/projected/fac108a2-b5b6-4241-82a2-e180d756608a-kube-api-access-j2wzr\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.368337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.368207 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-secret-metrics-server-tls\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.368337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.368310 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fac108a2-b5b6-4241-82a2-e180d756608a-metrics-server-audit-profiles\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.469037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.468989 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-secret-metrics-server-client-certs\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.469233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.469062 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-client-ca-bundle\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.469233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.469094 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wzr\" (UniqueName: \"kubernetes.io/projected/fac108a2-b5b6-4241-82a2-e180d756608a-kube-api-access-j2wzr\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.469233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.469135 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-secret-metrics-server-tls\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.469233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.469180 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fac108a2-b5b6-4241-82a2-e180d756608a-metrics-server-audit-profiles\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.469233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.469218 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fac108a2-b5b6-4241-82a2-e180d756608a-audit-log\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.469516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.469285 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac108a2-b5b6-4241-82a2-e180d756608a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.470088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.470053 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fac108a2-b5b6-4241-82a2-e180d756608a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.470284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.470240 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fac108a2-b5b6-4241-82a2-e180d756608a-audit-log\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.470586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.470560 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fac108a2-b5b6-4241-82a2-e180d756608a-metrics-server-audit-profiles\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.472824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.472796 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-client-ca-bundle\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.473787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.473742 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-secret-metrics-server-client-certs\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.473787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.473775 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fac108a2-b5b6-4241-82a2-e180d756608a-secret-metrics-server-tls\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.494257 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.494213 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wzr\" (UniqueName: \"kubernetes.io/projected/fac108a2-b5b6-4241-82a2-e180d756608a-kube-api-access-j2wzr\") pod \"metrics-server-8474bc4858-594xw\" (UID: \"fac108a2-b5b6-4241-82a2-e180d756608a\") " pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.555987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.555943 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:23.867926 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:23.867895 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8474bc4858-594xw"] Oct 31 21:16:23.870779 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:23.870755 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac108a2_b5b6_4241_82a2_e180d756608a.slice/crio-dd92d483291aacae5a0a61eb4152f3d585273cfe00c4b558006e04ebf4bc7369 WatchSource:0}: Error finding container dd92d483291aacae5a0a61eb4152f3d585273cfe00c4b558006e04ebf4bc7369: Status 404 returned error can't find the container with id dd92d483291aacae5a0a61eb4152f3d585273cfe00c4b558006e04ebf4bc7369 Oct 31 21:16:24.143566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.143477 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" event={"ID":"26b055db-6639-407e-a248-c1339b966438","Type":"ContainerStarted","Data":"58c9653b10335b8a4b08ecc03e17a88abe26c2451b15424160c28b9e3a3b55e7"} Oct 31 21:16:24.143566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.143550 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" event={"ID":"26b055db-6639-407e-a248-c1339b966438","Type":"ContainerStarted","Data":"c5d1a90415264d3fd9cffd23c1c366f691454559a0623b10674c83e1de5b7d68"} Oct 31 21:16:24.143770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.143594 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" event={"ID":"26b055db-6639-407e-a248-c1339b966438","Type":"ContainerStarted","Data":"d7b93db2f9508aca92610e3f835252b9a80887999c97df295f3ab959e46763cd"} Oct 31 21:16:24.146711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.146677 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerStarted","Data":"e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5"} Oct 31 21:16:24.146861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.146757 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerStarted","Data":"d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3"} Oct 31 21:16:24.146861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.146796 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerStarted","Data":"6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2"} Oct 31 21:16:24.146861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.146835 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerStarted","Data":"6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876"} Oct 31 21:16:24.146983 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.146870 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerStarted","Data":"95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad"} Oct 31 21:16:24.147857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:24.147837 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" event={"ID":"fac108a2-b5b6-4241-82a2-e180d756608a","Type":"ContainerStarted","Data":"dd92d483291aacae5a0a61eb4152f3d585273cfe00c4b558006e04ebf4bc7369"} Oct 31 21:16:25.155877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.155837 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" event={"ID":"26b055db-6639-407e-a248-c1339b966438","Type":"ContainerStarted","Data":"ff6fffaf0ee3e47a88b9fb6432a4cc46f415cdeba57b35280b678065cf1f2116"} Oct 31 21:16:25.156350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.155929 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" event={"ID":"26b055db-6639-407e-a248-c1339b966438","Type":"ContainerStarted","Data":"7182735a04dab7f330235a21e17121d15cf53bc8e1240298ae6c552b1dbc59ef"} Oct 31 21:16:25.156350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.155976 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" event={"ID":"26b055db-6639-407e-a248-c1339b966438","Type":"ContainerStarted","Data":"1562f3ff04d710836602cb4762f677de3a5abbd978fdbb2a3ae9466bb761e254"} Oct 31 21:16:25.156350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.156022 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:25.159683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.159654 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerStarted","Data":"c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb"} Oct 31 21:16:25.223134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.223074 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" podStartSLOduration=2.201099432 podStartE2EDuration="5.223060155s" totalImagesPullingTime="3.021960723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:25.210510252 +0000 UTC m=+25.335617497" watchObservedRunningTime="2026-10-31 21:16:25.223060155 +0000 UTC m=+25.348167386" Oct 31 21:16:25.272693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.272647 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.123901136 podStartE2EDuration="6.272633036s" totalImagesPullingTime="3.133553375s" totalInitContainerRuntime="15.178525ms" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:25.271785481 +0000 UTC m=+25.396892731" watchObservedRunningTime="2026-10-31 21:16:25.272633036 +0000 UTC m=+25.397740269" Oct 31 21:16:25.631903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.631871 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:16:25.635638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.635613 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.676079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676039 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 21:16:25.676251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676161 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 21:16:25.676251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676197 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9nvd4\"" Oct 31 21:16:25.676251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676194 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 21:16:25.676251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676245 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 21:16:25.676476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676289 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 21:16:25.676476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676318 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 21:16:25.676476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676428 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 21:16:25.676476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676450 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 21:16:25.676655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676493 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ffkaar1t4b0r3\"" Oct 31 21:16:25.676655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676505 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 21:16:25.676764 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676716 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 21:16:25.676818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.676784 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 21:16:25.677183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.677165 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 21:16:25.678120 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.678103 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:16:25.685873 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.685842 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.685991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.685942 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.685991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.685978 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686061 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686135 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686222 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686183 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686246 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686272 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686361 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686461 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686482 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686502 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686586 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686674 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686768 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686695 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686768 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686720 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686801 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686893 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.686963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.686936 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6bw4\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-kube-api-access-k6bw4\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.687052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.687032 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.751438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.751364 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 21:16:25.751438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.751364 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 21:16:25.788033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.787994 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788033 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788065 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788092 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788111 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788130 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788159 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788184 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788216 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788245 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788269 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788296 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788342 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788370 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788390 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788407 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788422 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.788740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788459 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6bw4\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-kube-api-access-k6bw4\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.789099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.788848 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.789099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.789056 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.789351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.789271 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.789433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.789358 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.789433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.789402 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.789883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.789561 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.791557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.791507 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.792101 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.792072 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.792465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.792446 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-web-config\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.793784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.792834 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.793784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.793045 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.793784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.793247 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.793784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.793454 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.793784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.793522 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.793784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.793773 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config-out\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.794128 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.794003 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.794202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.794178 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.795734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.795704 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.796049 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.796026 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.799637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.799612 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.799999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.799983 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.813904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.813872 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6bw4\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-kube-api-access-k6bw4\") pod \"prometheus-k8s-0\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:25.961971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:25.961870 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:26.164123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:26.164082 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" event={"ID":"fac108a2-b5b6-4241-82a2-e180d756608a","Type":"ContainerStarted","Data":"07e2e40d62520389be00a92fc9fbd429ded0a145ba75e2cc7eccdc1dbe0327f2"} Oct 31 21:16:26.207841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:26.207792 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" podStartSLOduration=1.840288371 podStartE2EDuration="3.207776432s" totalImagesPullingTime="1.367488061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:26.207602964 +0000 UTC m=+26.332710231" watchObservedRunningTime="2026-10-31 21:16:26.207776432 +0000 UTC m=+26.332883661" Oct 31 21:16:26.288797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:26.288713 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:16:26.290950 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:16:26.290918 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0dd759_8e86_4d90_98ef_852e02f2b3b8.slice/crio-97f459cac0034517fd28cdb70d9aaa9886184194a71aa6854f52b94585f7f2bf WatchSource:0}: Error finding container 97f459cac0034517fd28cdb70d9aaa9886184194a71aa6854f52b94585f7f2bf: Status 404 returned error can't find the container with id 97f459cac0034517fd28cdb70d9aaa9886184194a71aa6854f52b94585f7f2bf Oct 31 21:16:27.169116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:27.169077 4294 generic.go:356] "Generic (PLEG): container finished" podID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerID="ba9655f24a56744be33cae23d51b34329f1108eb83bb545c457815e4ed59286b" exitCode=0 Oct 31 21:16:27.169503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:27.169157 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"ba9655f24a56744be33cae23d51b34329f1108eb83bb545c457815e4ed59286b"} Oct 31 21:16:27.169503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:27.169228 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerStarted","Data":"97f459cac0034517fd28cdb70d9aaa9886184194a71aa6854f52b94585f7f2bf"} Oct 31 21:16:31.171872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.171843 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-555f8d6d4b-prhz2" Oct 31 21:16:31.195334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.195289 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerStarted","Data":"6634504cfbe31419593fabaaea788826be40eb1a68ce2f1a72e1ce29e7ee34c3"} Oct 31 21:16:31.195473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.195362 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerStarted","Data":"a796a357be56293cc006e0142f863057a6cc1adfe10075120883b4426d97e09f"} Oct 31 21:16:31.195473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.195389 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerStarted","Data":"2045b61908cd67203eafc196971ae17f2ebd2a258811baecc2707830dcf4299a"} Oct 31 21:16:31.195473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.195413 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerStarted","Data":"e27b909fa65b33c1aa7ca49986088badc4e1626bc4b6398bcd539f58bd628182"} Oct 31 21:16:31.195473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.195444 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerStarted","Data":"ea5b3531edc5b7a4cf023ecd006b3c1651424d752ed55b9f84e2be38d9c78c3b"} Oct 31 21:16:31.195473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.195467 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerStarted","Data":"b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e"} Oct 31 21:16:31.279524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:31.279422 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.962293137 podStartE2EDuration="6.279408048s" totalImagesPullingTime="3.302118044s" totalInitContainerRuntime="14.996867ms" isStatefulPod=false podCreationTimestamp="2026-10-31 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:16:31.278211985 +0000 UTC m=+31.403319222" watchObservedRunningTime="2026-10-31 21:16:31.279408048 +0000 UTC m=+31.404515309" Oct 31 21:16:31.309921 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:16:31.309885 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:16:32.084593 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:32.084563 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c59468d77-4655d" Oct 31 21:16:35.962501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:35.962450 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:16:36.134216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:36.134187 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z2f89" Oct 31 21:16:43.556062 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:43.555962 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:16:43.556679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:16:43.556337 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:17:01.309628 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:01.309592 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:17:03.561218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:03.561187 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:17:03.564938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:03.564918 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8474bc4858-594xw" Oct 31 21:17:20.615052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:20.615014 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:20.634192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:20.634160 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:20.978270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:20.978190 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:30.623083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:30.623045 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 21:17:30.623820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:30.623403 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy" containerID="cri-o://d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3" gracePeriod=120 Oct 31 21:17:30.623820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:30.623434 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="prom-label-proxy" containerID="cri-o://c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb" gracePeriod=120 Oct 31 21:17:30.623820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:30.623462 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-metric" containerID="cri-o://e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5" gracePeriod=120 Oct 31 21:17:30.623820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:30.623479 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-web" containerID="cri-o://6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2" gracePeriod=120 Oct 31 21:17:30.623820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:30.623398 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="alertmanager" containerID="cri-o://95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad" gracePeriod=120 Oct 31 21:17:30.624098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:30.623517 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="config-reloader" containerID="cri-o://6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876" gracePeriod=120 Oct 31 21:17:31.309979 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:31.309937 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:17:31.409166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409133 4294 generic.go:356] "Generic (PLEG): container finished" podID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerID="c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb" exitCode=0 Oct 31 21:17:31.409166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409158 4294 generic.go:356] "Generic (PLEG): container finished" podID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerID="e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5" exitCode=0 Oct 31 21:17:31.409166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409167 4294 generic.go:356] "Generic (PLEG): container finished" podID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerID="d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3" exitCode=0 Oct 31 21:17:31.409166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409174 4294 generic.go:356] "Generic (PLEG): container finished" podID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerID="6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876" exitCode=0 Oct 31 21:17:31.409431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409179 4294 generic.go:356] "Generic (PLEG): container finished" podID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerID="95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad" exitCode=0 Oct 31 21:17:31.409431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409203 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb"} Oct 31 21:17:31.409431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409264 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5"} Oct 31 21:17:31.409431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409292 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3"} Oct 31 21:17:31.409431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409329 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876"} Oct 31 21:17:31.409431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.409353 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad"} Oct 31 21:17:31.869719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.869695 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:17:31.947716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.947604 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-web-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-web-config\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.947716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.947680 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-kube-api-access-6w775\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-kube-api-access-6w775\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.947953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.947781 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-main-tls\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.947953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.947823 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-cluster-tls-config\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.947953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.947868 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-tls-assets\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-tls-assets\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948426 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948340 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-trusted-ca-bundle\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948426 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948390 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948467 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948524 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-metrics-client-ca\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948580 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-out\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948603 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-web\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948634 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-volume\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-volume\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.948769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.948677 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-main-db\") pod \"008e71ad-8d93-4da1-81fc-0e9447810f1f\" (UID: \"008e71ad-8d93-4da1-81fc-0e9447810f1f\") " Oct 31 21:17:31.949416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.949072 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-trusted-ca-bundle" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:31.949546 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.949522 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-main-db" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:17:31.949700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.949678 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-metrics-client-ca" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:31.950589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.950561 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-tls-assets" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:17:31.950941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.950920 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-kube-api-access-6w775" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "kube-api-access-6w775". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:17:31.951748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.951714 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-out" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:17:31.952083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.952056 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-web" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:31.952199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.952104 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:31.952328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.952279 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-main-tls" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:31.952978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.952956 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-metric" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:31.953271 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.953250 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-volume" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:31.957280 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.957252 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-cluster-tls-config" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:31.965785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:31.965749 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-web-config" pod "008e71ad-8d93-4da1-81fc-0e9447810f1f" (UID: "008e71ad-8d93-4da1-81fc-0e9447810f1f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:32.049662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049621 4294 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049653 4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049665 4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049675 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/008e71ad-8d93-4da1-81fc-0e9447810f1f-metrics-client-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049685 4294 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-out\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049694 4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049712 4294 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049720 4294 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/008e71ad-8d93-4da1-81fc-0e9447810f1f-alertmanager-main-db\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049728 4294 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-web-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049736 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w775\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-kube-api-access-6w775\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049744 4294 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-secret-alertmanager-main-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049752 4294 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008e71ad-8d93-4da1-81fc-0e9447810f1f-cluster-tls-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.049891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.049759 4294 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008e71ad-8d93-4da1-81fc-0e9447810f1f-tls-assets\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:32.416585 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.416540 4294 generic.go:356] "Generic (PLEG): container finished" podID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerID="6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2" exitCode=0 Oct 31 21:17:32.416783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.416634 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 21:17:32.416783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.416623 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2"} Oct 31 21:17:32.416972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.416785 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"008e71ad-8d93-4da1-81fc-0e9447810f1f","Type":"ContainerDied","Data":"2babd457e0bee5bf88c86bbbc8267f0dbed82f6b5a81f32eeacd67d3163b55c3"} Oct 31 21:17:32.416972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.416845 4294 scope.go:122] "RemoveContainer" containerID="c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb" Oct 31 21:17:32.425512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.425490 4294 scope.go:122] "RemoveContainer" containerID="e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5" Oct 31 21:17:32.432897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.432875 4294 scope.go:122] "RemoveContainer" containerID="d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3" Oct 31 21:17:32.439891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.439869 4294 scope.go:122] "RemoveContainer" containerID="6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2" Oct 31 21:17:32.446433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.446415 4294 scope.go:122] "RemoveContainer" containerID="6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876" Oct 31 21:17:32.453372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.453348 4294 scope.go:122] "RemoveContainer" containerID="95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad" Oct 31 21:17:32.459885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.459868 4294 scope.go:122] "RemoveContainer" containerID="93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8" Oct 31 21:17:32.466314 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.466281 4294 scope.go:122] "RemoveContainer" containerID="c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb" Oct 31 21:17:32.466578 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:32.466558 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb\": container with ID starting with c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb not found: ID does not exist" containerID="c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb" Oct 31 21:17:32.466636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.466588 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb"} err="failed to get container status \"c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb\": rpc error: code = NotFound desc = could not find container \"c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb\": container with ID starting with c7e6e6ec979d67b89327cc5dd6599af830ae20b4bcdeed5b18e32182e36fe0fb not found: ID does not exist" Oct 31 21:17:32.466636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.466619 4294 scope.go:122] "RemoveContainer" containerID="e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5" Oct 31 21:17:32.466879 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:32.466861 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5\": container with ID starting with e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5 not found: ID does not exist" containerID="e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5" Oct 31 21:17:32.466944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.466891 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5"} err="failed to get container status \"e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5\": rpc error: code = NotFound desc = could not find container \"e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5\": container with ID starting with e25bc180bb46a83dcc57ea62921c47d48ff165cf8e2c6a47e81185d6d96ad1a5 not found: ID does not exist" Oct 31 21:17:32.466944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.466914 4294 scope.go:122] "RemoveContainer" containerID="d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3" Oct 31 21:17:32.467166 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:32.467149 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3\": container with ID starting with d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3 not found: ID does not exist" containerID="d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3" Oct 31 21:17:32.467204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.467173 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3"} err="failed to get container status \"d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3\": rpc error: code = NotFound desc = could not find container \"d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3\": container with ID starting with d4c3b784e2d1bc75482192faf920d668cb3ffdc82551e62ec74b0b8251d8a1e3 not found: ID does not exist" Oct 31 21:17:32.467204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.467188 4294 scope.go:122] "RemoveContainer" containerID="6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2" Oct 31 21:17:32.467454 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:32.467435 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2\": container with ID starting with 6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2 not found: ID does not exist" containerID="6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2" Oct 31 21:17:32.467530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.467460 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2"} err="failed to get container status \"6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2\": rpc error: code = NotFound desc = could not find container \"6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2\": container with ID starting with 6d6fbdcd77c7556626cdef4fad94914cd618a1ad6f2729df32d7695c6036bec2 not found: ID does not exist" Oct 31 21:17:32.467530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.467481 4294 scope.go:122] "RemoveContainer" containerID="6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876" Oct 31 21:17:32.467809 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:32.467792 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876\": container with ID starting with 6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876 not found: ID does not exist" containerID="6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876" Oct 31 21:17:32.467845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.467824 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876"} err="failed to get container status \"6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876\": rpc error: code = NotFound desc = could not find container \"6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876\": container with ID starting with 6bc5744e3723d4993ef77442c14088fc4f72f12ba9d6cb19fbcedd6e6b341876 not found: ID does not exist" Oct 31 21:17:32.467845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.467838 4294 scope.go:122] "RemoveContainer" containerID="95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad" Oct 31 21:17:32.468054 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:32.468038 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad\": container with ID starting with 95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad not found: ID does not exist" containerID="95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad" Oct 31 21:17:32.468117 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.468060 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad"} err="failed to get container status \"95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad\": rpc error: code = NotFound desc = could not find container \"95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad\": container with ID starting with 95cf37dd5f1ec260befecda0b86f08609ef58e159f94012bf5834137ee04b5ad not found: ID does not exist" Oct 31 21:17:32.468117 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.468081 4294 scope.go:122] "RemoveContainer" containerID="93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8" Oct 31 21:17:32.468340 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:32.468323 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8\": container with ID starting with 93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8 not found: ID does not exist" containerID="93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8" Oct 31 21:17:32.468382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.468344 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8"} err="failed to get container status \"93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8\": rpc error: code = NotFound desc = could not find container \"93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8\": container with ID starting with 93e13df6a777597dea218067a1fdd6f2ba72a0e798a10cea1d96c87173229af8 not found: ID does not exist" Oct 31 21:17:32.485476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.485450 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 21:17:32.497525 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:32.497496 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 21:17:33.981986 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:33.981952 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" path="/var/lib/kubelet/pods/008e71ad-8d93-4da1-81fc-0e9447810f1f/volumes" Oct 31 21:17:35.307271 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.307235 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:17:35.307724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.307544 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="prometheus" containerID="cri-o://b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e" gracePeriod=600 Oct 31 21:17:35.307724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.307560 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy" containerID="cri-o://a796a357be56293cc006e0142f863057a6cc1adfe10075120883b4426d97e09f" gracePeriod=600 Oct 31 21:17:35.307724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.307594 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="thanos-sidecar" containerID="cri-o://e27b909fa65b33c1aa7ca49986088badc4e1626bc4b6398bcd539f58bd628182" gracePeriod=600 Oct 31 21:17:35.307724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.307618 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6634504cfbe31419593fabaaea788826be40eb1a68ce2f1a72e1ce29e7ee34c3" gracePeriod=600 Oct 31 21:17:35.307724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.307611 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="config-reloader" containerID="cri-o://ea5b3531edc5b7a4cf023ecd006b3c1651424d752ed55b9f84e2be38d9c78c3b" gracePeriod=600 Oct 31 21:17:35.307724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.307640 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-web" containerID="cri-o://2045b61908cd67203eafc196971ae17f2ebd2a258811baecc2707830dcf4299a" gracePeriod=600 Oct 31 21:17:35.436210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436179 4294 generic.go:356] "Generic (PLEG): container finished" podID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerID="6634504cfbe31419593fabaaea788826be40eb1a68ce2f1a72e1ce29e7ee34c3" exitCode=0 Oct 31 21:17:35.436210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436204 4294 generic.go:356] "Generic (PLEG): container finished" podID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerID="a796a357be56293cc006e0142f863057a6cc1adfe10075120883b4426d97e09f" exitCode=0 Oct 31 21:17:35.436210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436210 4294 generic.go:356] "Generic (PLEG): container finished" podID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerID="e27b909fa65b33c1aa7ca49986088badc4e1626bc4b6398bcd539f58bd628182" exitCode=0 Oct 31 21:17:35.436210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436216 4294 generic.go:356] "Generic (PLEG): container finished" podID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerID="ea5b3531edc5b7a4cf023ecd006b3c1651424d752ed55b9f84e2be38d9c78c3b" exitCode=0 Oct 31 21:17:35.436559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436221 4294 generic.go:356] "Generic (PLEG): container finished" podID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerID="b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e" exitCode=0 Oct 31 21:17:35.436559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436251 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"6634504cfbe31419593fabaaea788826be40eb1a68ce2f1a72e1ce29e7ee34c3"} Oct 31 21:17:35.436559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436375 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"a796a357be56293cc006e0142f863057a6cc1adfe10075120883b4426d97e09f"} Oct 31 21:17:35.436559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436415 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"e27b909fa65b33c1aa7ca49986088badc4e1626bc4b6398bcd539f58bd628182"} Oct 31 21:17:35.436559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436457 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"ea5b3531edc5b7a4cf023ecd006b3c1651424d752ed55b9f84e2be38d9c78c3b"} Oct 31 21:17:35.436559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.436496 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e"} Oct 31 21:17:35.962397 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:35.962349 4294 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 21:17:35.962609 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:35.962419 4294 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 21:17:35.962609 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:17:35.962445 4294 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 21:17:35.962609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:35.962466 4294 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 31 21:17:36.444235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.444201 4294 generic.go:356] "Generic (PLEG): container finished" podID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerID="2045b61908cd67203eafc196971ae17f2ebd2a258811baecc2707830dcf4299a" exitCode=0 Oct 31 21:17:36.444614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.444278 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"2045b61908cd67203eafc196971ae17f2ebd2a258811baecc2707830dcf4299a"} Oct 31 21:17:36.558668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.558640 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:36.684520 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684432 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-db\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684520 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684474 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-tls-assets\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684520 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684509 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-kube-rbac-proxy\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684544 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684576 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-trusted-ca-bundle\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684612 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-1\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684647 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-web-config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-web-config\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684677 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684707 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-2\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684739 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.684800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684781 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-thanos-prometheus-http-client-file\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684808 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-metrics-client-ca\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684848 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-0\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684880 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config-out\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684938 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-kube-api-access-k6bw4\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-kube-api-access-k6bw4\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.684967 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-tls\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685013 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-metrics-client-certs\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685043 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-grpc-tls\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685072 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-kubelet-serving-ca-bundle\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685123 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-serving-certs-ca-bundle\") pod \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\" (UID: \"cd0dd759-8e86-4d90-98ef-852e02f2b3b8\") " Oct 31 21:17:36.685206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685167 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-2" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:36.685714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685559 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-serving-certs-ca-bundle" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:36.685714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685568 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-trusted-ca-bundle" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:36.686502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.685949 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-metrics-client-ca" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:36.686502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.686036 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-db" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:17:36.686502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.686192 4294 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.686502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.686217 4294 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-db\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.686502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.686235 4294 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.686502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.686251 4294 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.686502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.686267 4294 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-metrics-client-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.687094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.687068 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-1" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:36.687278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.687188 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-kube-rbac-proxy" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.687675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.687475 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-kubelet-serving-ca-bundle" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:36.687875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.687847 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-0" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:17:36.688264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.688233 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.688973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.688934 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-tls-assets" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:17:36.689151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.689122 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-tls" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.690275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.690238 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-metrics-client-certs" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.690275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.690260 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-grpc-tls" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.690469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.690361 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-kube-rbac-proxy-web" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.690469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.690437 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config-out" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:17:36.690579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.690507 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-kube-api-access-k6bw4" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "kube-api-access-k6bw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:17:36.690634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.690574 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-thanos-prometheus-http-client-file" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.691620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.691600 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-thanos-sidecar-tls" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.700636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.700597 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-web-config" pod "cd0dd759-8e86-4d90-98ef-852e02f2b3b8" (UID: "cd0dd759-8e86-4d90-98ef-852e02f2b3b8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:17:36.787151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787104 4294 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787162 4294 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-web-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787175 4294 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787187 4294 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787197 4294 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787207 4294 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787216 4294 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config-out\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787235 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6bw4\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-kube-api-access-k6bw4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787245 4294 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787253 4294 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-metrics-client-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787262 4294 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-grpc-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787270 4294 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787278 4294 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-tls-assets\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787287 4294 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-secret-kube-rbac-proxy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:36.787352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:36.787295 4294 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cd0dd759-8e86-4d90-98ef-852e02f2b3b8-config\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:17:37.451047 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.451010 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cd0dd759-8e86-4d90-98ef-852e02f2b3b8","Type":"ContainerDied","Data":"97f459cac0034517fd28cdb70d9aaa9886184194a71aa6854f52b94585f7f2bf"} Oct 31 21:17:37.451454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.451093 4294 scope.go:122] "RemoveContainer" containerID="6634504cfbe31419593fabaaea788826be40eb1a68ce2f1a72e1ce29e7ee34c3" Oct 31 21:17:37.451454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.451027 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.458461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.458435 4294 scope.go:122] "RemoveContainer" containerID="a796a357be56293cc006e0142f863057a6cc1adfe10075120883b4426d97e09f" Oct 31 21:17:37.465029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.465008 4294 scope.go:122] "RemoveContainer" containerID="2045b61908cd67203eafc196971ae17f2ebd2a258811baecc2707830dcf4299a" Oct 31 21:17:37.471235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.471212 4294 scope.go:122] "RemoveContainer" containerID="e27b909fa65b33c1aa7ca49986088badc4e1626bc4b6398bcd539f58bd628182" Oct 31 21:17:37.477360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.477339 4294 scope.go:122] "RemoveContainer" containerID="ea5b3531edc5b7a4cf023ecd006b3c1651424d752ed55b9f84e2be38d9c78c3b" Oct 31 21:17:37.483713 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.483694 4294 scope.go:122] "RemoveContainer" containerID="b11194c4eb5764b477f52d28fa7650ec35512a1c2b025441a1dfa5461b1d0a3e" Oct 31 21:17:37.490547 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.490529 4294 scope.go:122] "RemoveContainer" containerID="ba9655f24a56744be33cae23d51b34329f1108eb83bb545c457815e4ed59286b" Oct 31 21:17:37.524541 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.524513 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:17:37.536374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.536344 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:17:37.561067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561035 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:17:37.561288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561275 4294 cpu_manager.go:414] "containerMap: removing container" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="prom-label-proxy" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561290 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="prom-label-proxy" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561313 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-web" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561321 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-web" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561333 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="init-config-reloader" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561341 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="init-config-reloader" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561349 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561354 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy" Oct 31 21:17:37.561360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561362 4294 cpu_manager.go:414] "containerMap: removing container" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="init-config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561367 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="init-config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561373 4294 cpu_manager.go:414] "containerMap: removing container" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="alertmanager" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561377 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="alertmanager" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561385 4294 cpu_manager.go:414] "containerMap: removing container" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561390 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561396 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-thanos" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561400 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-thanos" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561412 4294 cpu_manager.go:414] "containerMap: removing container" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-web" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561417 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-web" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561429 4294 cpu_manager.go:414] "containerMap: removing container" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561433 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561439 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="prometheus" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561443 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="prometheus" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561449 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="thanos-sidecar" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561453 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="thanos-sidecar" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561460 4294 cpu_manager.go:414] "containerMap: removing container" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-metric" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561464 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-metric" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561472 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561475 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561518 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561526 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-thanos" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561533 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-web" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561538 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561543 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="prometheus" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561548 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="thanos-sidecar" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561553 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="kube-rbac-proxy-metric" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561559 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="config-reloader" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561564 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="prom-label-proxy" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561569 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy-web" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561573 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" containerName="kube-rbac-proxy" Oct 31 21:17:37.561586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.561578 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="008e71ad-8d93-4da1-81fc-0e9447810f1f" containerName="alertmanager" Oct 31 21:17:37.566658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.566640 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.582099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582065 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 21:17:37.582296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582252 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 21:17:37.582296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582252 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 21:17:37.582449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582386 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 21:17:37.582523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582473 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 21:17:37.582592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582577 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 21:17:37.582681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582606 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 21:17:37.582681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582608 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 21:17:37.582681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582610 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 21:17:37.582681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582610 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 21:17:37.582912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582759 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 21:17:37.582912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582770 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ffkaar1t4b0r3\"" Oct 31 21:17:37.582912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582784 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 21:17:37.582912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.582813 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-9nvd4\"" Oct 31 21:17:37.595766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.595732 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 21:17:37.595766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.595737 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 21:17:37.596887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.596864 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:17:37.694689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694598 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.694689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694640 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.694689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694666 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.694901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694737 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jplq\" (UniqueName: \"kubernetes.io/projected/ddb09a2f-1e81-4837-937e-f50442ec2d03-kube-api-access-5jplq\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.694901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694790 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.694901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694811 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddb09a2f-1e81-4837-937e-f50442ec2d03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.694901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694835 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694922 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.694953 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695019 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695040 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695062 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695081 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddb09a2f-1e81-4837-937e-f50442ec2d03-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695129 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695151 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695207 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695245 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695321 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-config\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695340 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.695415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.695400 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.795785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795737 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.795785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795783 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795803 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795826 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddb09a2f-1e81-4837-937e-f50442ec2d03-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795850 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795866 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795886 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795913 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795941 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-config\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.795967 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796000 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796063 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796085 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796110 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796139 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jplq\" (UniqueName: \"kubernetes.io/projected/ddb09a2f-1e81-4837-937e-f50442ec2d03-kube-api-access-5jplq\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796205 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796236 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddb09a2f-1e81-4837-937e-f50442ec2d03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796257 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796287 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796340 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796525 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.796940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.796689 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.797363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.797335 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.797474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.797371 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.798095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.797983 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.800879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.798438 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.800879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.800534 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-web-config\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.800879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.800838 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.800879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.800846 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ddb09a2f-1e81-4837-937e-f50442ec2d03-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.800879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.800860 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ddb09a2f-1e81-4837-937e-f50442ec2d03-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801028 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801029 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801211 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ddb09a2f-1e81-4837-937e-f50442ec2d03-config-out\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801330 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801464 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801605 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-config\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801692 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.801763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.801709 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.802662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.802644 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ddb09a2f-1e81-4837-937e-f50442ec2d03-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.826105 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.826079 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jplq\" (UniqueName: \"kubernetes.io/projected/ddb09a2f-1e81-4837-937e-f50442ec2d03-kube-api-access-5jplq\") pod \"prometheus-k8s-0\" (UID: \"ddb09a2f-1e81-4837-937e-f50442ec2d03\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.890623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.890584 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:17:37.982027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:37.981956 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0dd759-8e86-4d90-98ef-852e02f2b3b8" path="/var/lib/kubelet/pods/cd0dd759-8e86-4d90-98ef-852e02f2b3b8/volumes" Oct 31 21:17:38.216566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:38.216539 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 21:17:38.217942 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:17:38.217918 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb09a2f_1e81_4837_937e_f50442ec2d03.slice/crio-a13c20a271ca2f69d2b0387994ecd57e1c543be8502859a9aa253501e4768769 WatchSource:0}: Error finding container a13c20a271ca2f69d2b0387994ecd57e1c543be8502859a9aa253501e4768769: Status 404 returned error can't find the container with id a13c20a271ca2f69d2b0387994ecd57e1c543be8502859a9aa253501e4768769 Oct 31 21:17:38.458678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:38.458636 4294 generic.go:356] "Generic (PLEG): container finished" podID="ddb09a2f-1e81-4837-937e-f50442ec2d03" containerID="e36281c933d039fa49935b1371a0aed6ddc9b41099c607266b50ed366e90e719" exitCode=0 Oct 31 21:17:38.459156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:38.458726 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerDied","Data":"e36281c933d039fa49935b1371a0aed6ddc9b41099c607266b50ed366e90e719"} Oct 31 21:17:38.459156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:38.458856 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerStarted","Data":"a13c20a271ca2f69d2b0387994ecd57e1c543be8502859a9aa253501e4768769"} Oct 31 21:17:39.466126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:39.466090 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerStarted","Data":"4d1c464b46e23616cbb7f9769e93d79fdf3922c769250cdb60e686c66237a505"} Oct 31 21:17:39.466569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:39.466167 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerStarted","Data":"ee30498f5719446de3c7916a7793e9b4afc6abc6b6a86448875b6f118ff0b3d1"} Oct 31 21:17:39.466569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:39.466205 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerStarted","Data":"f162d61fcb92c520bb60b7b3e68a5f10404df5d761b3cfc27aa6fbcfe528d59c"} Oct 31 21:17:39.466569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:39.466250 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerStarted","Data":"b5725149f2316db95fee41cd3be5041d6e7e24450ba2dcad134460982cc478dd"} Oct 31 21:17:39.466569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:39.466292 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerStarted","Data":"97289ab5a758d745acebb24d76622519b82c280f84a761a3a73e91846ed46145"} Oct 31 21:17:39.466569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:39.466347 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ddb09a2f-1e81-4837-937e-f50442ec2d03","Type":"ContainerStarted","Data":"07c533caa6e19dec69da35603ade29ee68186cda1febcfd7b2e05738adc64451"} Oct 31 21:17:39.539623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:39.539562 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.524213777 podStartE2EDuration="2.539542271s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.328494ms" isStatefulPod=false podCreationTimestamp="2026-10-31 21:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:17:39.538765355 +0000 UTC m=+99.663872586" watchObservedRunningTime="2026-10-31 21:17:39.539542271 +0000 UTC m=+99.664649504" Oct 31 21:17:42.891740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:17:42.891700 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:18:01.308902 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:18:01.308864 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:18:31.308522 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:18:31.308477 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:18:37.891174 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:18:37.891130 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:18:37.907038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:18:37.907015 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:18:38.676791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:18:38.676761 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 21:19:01.309219 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:19:01.309181 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:19:31.309108 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:19:31.309062 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:20:01.308474 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:20:01.308390 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:20:31.308748 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:20:31.308716 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:20:34.036932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.036901 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs"] Oct 31 21:20:34.039558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.039543 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.055693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.055674 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 21:20:34.055794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.055733 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 21:20:34.055794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.055733 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dn2rx\"" Oct 31 21:20:34.055863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.055801 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 21:20:34.069144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.069120 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs"] Oct 31 21:20:34.169727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.169698 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b"] Oct 31 21:20:34.172805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.172783 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e8c40-6b3c-4c53-8476-e527511c7340-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.172900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.172826 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.172900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.172844 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.172900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.172893 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.173055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.172951 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e8c40-6b3c-4c53-8476-e527511c7340-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.173055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.172993 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.173132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.173103 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.173204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.173185 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.173254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.173218 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsmc\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-kube-api-access-dlsmc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.173288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.173252 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.188061 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.188038 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-qlnq9\"" Oct 31 21:20:34.195369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.195340 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b"] Oct 31 21:20:34.274247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274212 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e8c40-6b3c-4c53-8476-e527511c7340-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274252 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.274411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274279 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.274411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274358 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245bacec-3301-4fc3-afb3-b5795bc24233-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.274532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274448 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245bacec-3301-4fc3-afb3-b5795bc24233-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.274532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274477 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwt2l\" (UniqueName: \"kubernetes.io/projected/245bacec-3301-4fc3-afb3-b5795bc24233-kube-api-access-wwt2l\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.274597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274557 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.274597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274584 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245bacec-3301-4fc3-afb3-b5795bc24233-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.274672 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274634 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274668 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274692 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e8c40-6b3c-4c53-8476-e527511c7340-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274719 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274871 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274774 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274871 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274802 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274871 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274825 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsmc\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-kube-api-access-dlsmc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.274871 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274851 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.275070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274880 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.275070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.274911 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.275070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.275026 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.275070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.275052 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.275254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.275069 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e8c40-6b3c-4c53-8476-e527511c7340-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.275254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.275099 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.275347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.275288 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.277407 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.277378 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.277517 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.277499 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e8c40-6b3c-4c53-8476-e527511c7340-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.300763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.300710 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.301002 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.300985 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsmc\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-kube-api-access-dlsmc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-gmgjs\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.351985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.351948 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:34.376279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376246 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245bacec-3301-4fc3-afb3-b5795bc24233-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.376388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376293 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwt2l\" (UniqueName: \"kubernetes.io/projected/245bacec-3301-4fc3-afb3-b5795bc24233-kube-api-access-wwt2l\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.376388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376355 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.376388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376380 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245bacec-3301-4fc3-afb3-b5795bc24233-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.376578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376550 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.376692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376602 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.376692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376656 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.377007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376681 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.377102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.377031 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245bacec-3301-4fc3-afb3-b5795bc24233-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.377102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376959 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.377102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376783 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.377102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.376976 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.377374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.377097 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.377431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.377395 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/245bacec-3301-4fc3-afb3-b5795bc24233-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.378679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.378661 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/245bacec-3301-4fc3-afb3-b5795bc24233-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.378992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.378976 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/245bacec-3301-4fc3-afb3-b5795bc24233-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.401259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.400617 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwt2l\" (UniqueName: \"kubernetes.io/projected/245bacec-3301-4fc3-afb3-b5795bc24233-kube-api-access-wwt2l\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.401527 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.401485 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/245bacec-3301-4fc3-afb3-b5795bc24233-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b\" (UID: \"245bacec-3301-4fc3-afb3-b5795bc24233\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.488784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.488755 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:34.661947 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.661925 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs"] Oct 31 21:20:34.664039 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:20:34.664011 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273e8c40_6b3c_4c53_8476_e527511c7340.slice/crio-14bddf84790876d062b750253277dfae4a33a97855f85dbf44f41e6092a17c2d WatchSource:0}: Error finding container 14bddf84790876d062b750253277dfae4a33a97855f85dbf44f41e6092a17c2d: Status 404 returned error can't find the container with id 14bddf84790876d062b750253277dfae4a33a97855f85dbf44f41e6092a17c2d Oct 31 21:20:34.785315 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:34.785279 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b"] Oct 31 21:20:34.786773 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:20:34.786748 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245bacec_3301_4fc3_afb3_b5795bc24233.slice/crio-ab5c3ce5e59e795248810aacd1f60f6c782a4fe3cbdae8fd6f3744f23a76266d WatchSource:0}: Error finding container ab5c3ce5e59e795248810aacd1f60f6c782a4fe3cbdae8fd6f3744f23a76266d: Status 404 returned error can't find the container with id ab5c3ce5e59e795248810aacd1f60f6c782a4fe3cbdae8fd6f3744f23a76266d Oct 31 21:20:35.027656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:35.027619 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" event={"ID":"245bacec-3301-4fc3-afb3-b5795bc24233","Type":"ContainerStarted","Data":"ab5c3ce5e59e795248810aacd1f60f6c782a4fe3cbdae8fd6f3744f23a76266d"} Oct 31 21:20:35.028761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:35.028739 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" event={"ID":"273e8c40-6b3c-4c53-8476-e527511c7340","Type":"ContainerStarted","Data":"14bddf84790876d062b750253277dfae4a33a97855f85dbf44f41e6092a17c2d"} Oct 31 21:20:37.367202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:37.367166 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:20:37.367579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:37.367243 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:20:37.367579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:37.367287 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:20:37.524480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:37.524450 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:20:37.524575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:37.524516 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:20:37.524575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:37.524551 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:20:38.041572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.041531 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" event={"ID":"245bacec-3301-4fc3-afb3-b5795bc24233","Type":"ContainerStarted","Data":"38c6abe5155aa188330d5cd9838d5178259ceb21ccd6f36c60bb2bcfba1d7717"} Oct 31 21:20:38.042991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.042968 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" event={"ID":"273e8c40-6b3c-4c53-8476-e527511c7340","Type":"ContainerStarted","Data":"af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741"} Oct 31 21:20:38.100264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.100215 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" podStartSLOduration=1.365079847 podStartE2EDuration="4.100198222s" totalImagesPullingTime="2.735118375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:20:38.10007279 +0000 UTC m=+278.225180041" watchObservedRunningTime="2026-10-31 21:20:38.100198222 +0000 UTC m=+278.225305452" Oct 31 21:20:38.149375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.149331 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podStartSLOduration=1.4488324719999999 podStartE2EDuration="4.149319527s" totalImagesPullingTime="2.700487055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:20:38.148176027 +0000 UTC m=+278.273283258" watchObservedRunningTime="2026-10-31 21:20:38.149319527 +0000 UTC m=+278.274426758" Oct 31 21:20:38.352981 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.352907 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:38.356932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.356914 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:38.489965 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.489937 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:38.494008 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:38.493984 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:39.046570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:39.046525 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:39.046782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:39.046734 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:39.047630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:39.047602 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:20:39.047750 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:39.047661 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b" Oct 31 21:20:59.980244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:59.980212 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:20:59.980776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:20:59.980334 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:21:01.308983 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:21:01.308952 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:21:12.069095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.069062 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g"] Oct 31 21:21:12.078220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.078200 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.108244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.108211 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g"] Oct 31 21:21:12.152500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.152467 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-webhook-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.152630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.152507 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-apiservice-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.152630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.152527 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2216de4b-c17b-4efd-bf45-5667a2cc4084-manifests\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.152746 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.152704 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-metrics-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.152788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.152767 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2216de4b-c17b-4efd-bf45-5667a2cc4084-tmp\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.152823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.152807 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnqhz\" (UniqueName: \"kubernetes.io/projected/2216de4b-c17b-4efd-bf45-5667a2cc4084-kube-api-access-lnqhz\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.160456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.160435 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 21:21:12.160456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.160444 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 21:21:12.160779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.160761 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-lmxhd\"" Oct 31 21:21:12.160826 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.160779 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 21:21:12.160826 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.160809 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 21:21:12.161112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.161100 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 21:21:12.253376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.253352 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-metrics-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.253497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.253383 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2216de4b-c17b-4efd-bf45-5667a2cc4084-tmp\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.253497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.253409 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnqhz\" (UniqueName: \"kubernetes.io/projected/2216de4b-c17b-4efd-bf45-5667a2cc4084-kube-api-access-lnqhz\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.253497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.253461 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-webhook-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.253497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.253491 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-apiservice-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.253704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.253520 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2216de4b-c17b-4efd-bf45-5667a2cc4084-manifests\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.253876 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.253854 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2216de4b-c17b-4efd-bf45-5667a2cc4084-manifests\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.255689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.255661 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2216de4b-c17b-4efd-bf45-5667a2cc4084-tmp\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.255899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.255880 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-webhook-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.255958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.255918 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-apiservice-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.256012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.255962 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2216de4b-c17b-4efd-bf45-5667a2cc4084-metrics-cert\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.278503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.278481 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnqhz\" (UniqueName: \"kubernetes.io/projected/2216de4b-c17b-4efd-bf45-5667a2cc4084-kube-api-access-lnqhz\") pod \"opendatahub-operator-controller-manager-899c649cb-dtn6g\" (UID: \"2216de4b-c17b-4efd-bf45-5667a2cc4084\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.392687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.392635 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:12.739342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.734551 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g"] Oct 31 21:21:12.739342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:12.737662 4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:21:13.169504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:13.169468 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" event={"ID":"2216de4b-c17b-4efd-bf45-5667a2cc4084","Type":"ContainerStarted","Data":"e0d33b82722610eca3f46c11804dbdfd55cf8258c491542737971cbb32ec71df"} Oct 31 21:21:16.181652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:16.181583 4294 generic.go:356] "Generic (PLEG): container finished" podID="2216de4b-c17b-4efd-bf45-5667a2cc4084" containerID="475dbdfe90bac6c4267c05921ebf66d0299b1aaea4e8ab60bda69fa792999315" exitCode=0 Oct 31 21:21:16.181652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:16.181645 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" event={"ID":"2216de4b-c17b-4efd-bf45-5667a2cc4084","Type":"ContainerDied","Data":"475dbdfe90bac6c4267c05921ebf66d0299b1aaea4e8ab60bda69fa792999315"} Oct 31 21:21:17.187360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:17.187321 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" event={"ID":"2216de4b-c17b-4efd-bf45-5667a2cc4084","Type":"ContainerStarted","Data":"409d501a2f4fa469b21a903765b950a3c87ec79736f2a5da26ff82966fb1c26f"} Oct 31 21:21:17.187850 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:17.187451 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:17.289009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:17.288962 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" podStartSLOduration=1.94144407 podStartE2EDuration="5.288947869s" totalImagesPullingTime="2.906968111s" totalInitContainerRuntime="440.535688ms" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:17.276611385 +0000 UTC m=+317.401718616" watchObservedRunningTime="2026-10-31 21:21:17.288947869 +0000 UTC m=+317.414055099" Oct 31 21:21:28.194349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:28.194316 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-899c649cb-dtn6g" Oct 31 21:21:29.434706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.434679 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np"] Oct 31 21:21:29.440904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.440887 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:29.450423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.450398 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np"] Oct 31 21:21:29.462798 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.462776 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 31 21:21:29.462883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.462817 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 31 21:21:29.462971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.462953 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-ctjr8\"" Oct 31 21:21:29.486557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.486521 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxhp\" (UniqueName: \"kubernetes.io/projected/f11f8609-6251-4ade-838f-d706cc303ad2-kube-api-access-wdxhp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cn7np\" (UID: \"f11f8609-6251-4ade-838f-d706cc303ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:29.486646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.486590 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11f8609-6251-4ade-838f-d706cc303ad2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cn7np\" (UID: \"f11f8609-6251-4ade-838f-d706cc303ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:29.587606 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.587584 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11f8609-6251-4ade-838f-d706cc303ad2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cn7np\" (UID: \"f11f8609-6251-4ade-838f-d706cc303ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:29.587701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.587637 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxhp\" (UniqueName: \"kubernetes.io/projected/f11f8609-6251-4ade-838f-d706cc303ad2-kube-api-access-wdxhp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cn7np\" (UID: \"f11f8609-6251-4ade-838f-d706cc303ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:29.587912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.587895 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f11f8609-6251-4ade-838f-d706cc303ad2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cn7np\" (UID: \"f11f8609-6251-4ade-838f-d706cc303ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:29.613113 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.613093 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxhp\" (UniqueName: \"kubernetes.io/projected/f11f8609-6251-4ade-838f-d706cc303ad2-kube-api-access-wdxhp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-cn7np\" (UID: \"f11f8609-6251-4ade-838f-d706cc303ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:29.752737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:29.752717 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" Oct 31 21:21:30.079246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.079222 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np"] Oct 31 21:21:30.080889 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:21:30.080865 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf11f8609_6251_4ade_838f_d706cc303ad2.slice/crio-407ef3997980495b0ccc015f872a7abe7218757b3de044a46191ed1d56011659 WatchSource:0}: Error finding container 407ef3997980495b0ccc015f872a7abe7218757b3de044a46191ed1d56011659: Status 404 returned error can't find the container with id 407ef3997980495b0ccc015f872a7abe7218757b3de044a46191ed1d56011659 Oct 31 21:21:30.232046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.232016 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" event={"ID":"f11f8609-6251-4ade-838f-d706cc303ad2","Type":"ContainerStarted","Data":"407ef3997980495b0ccc015f872a7abe7218757b3de044a46191ed1d56011659"} Oct 31 21:21:30.418991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.418929 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh"] Oct 31 21:21:30.423206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.423184 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" Oct 31 21:21:30.441183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.441155 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 21:21:30.454143 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.454121 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 31 21:21:30.454229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.454171 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 21:21:30.454229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.454175 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wnxh4\"" Oct 31 21:21:30.455393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.455375 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh"] Oct 31 21:21:30.494261 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.494239 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znbcx\" (UniqueName: \"kubernetes.io/projected/9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777-kube-api-access-znbcx\") pod \"dns-operator-controller-manager-8557cbdb84-774gh\" (UID: \"9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" Oct 31 21:21:30.595023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.594992 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znbcx\" (UniqueName: \"kubernetes.io/projected/9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777-kube-api-access-znbcx\") pod \"dns-operator-controller-manager-8557cbdb84-774gh\" (UID: \"9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" Oct 31 21:21:30.625888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.625860 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znbcx\" (UniqueName: \"kubernetes.io/projected/9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777-kube-api-access-znbcx\") pod \"dns-operator-controller-manager-8557cbdb84-774gh\" (UID: \"9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" Oct 31 21:21:30.735085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:30.735061 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" Oct 31 21:21:31.049110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:31.049036 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh"] Oct 31 21:21:31.052840 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:21:31.052810 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f8bd849_a9b6_4f4d_83c6_0bbb9ee0f777.slice/crio-b02dd8fa4190b744755e440ebae3bac19b9581c2d57b6131136b732b635cb208 WatchSource:0}: Error finding container b02dd8fa4190b744755e440ebae3bac19b9581c2d57b6131136b732b635cb208: Status 404 returned error can't find the container with id b02dd8fa4190b744755e440ebae3bac19b9581c2d57b6131136b732b635cb208 Oct 31 21:21:31.240139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:31.240101 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" event={"ID":"9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777","Type":"ContainerStarted","Data":"b02dd8fa4190b744755e440ebae3bac19b9581c2d57b6131136b732b635cb208"} Oct 31 21:21:31.315069 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:21:31.314986 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:21:33.249528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:33.249491 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" event={"ID":"f11f8609-6251-4ade-838f-d706cc303ad2","Type":"ContainerStarted","Data":"0a3dfb5e34ee6bd0728fff472fb2c107bf1c0a41a75b3e308bcaa2f494a3f999"} Oct 31 21:21:33.327036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:33.326978 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-cn7np" podStartSLOduration=1.8047002650000001 podStartE2EDuration="4.32696609s" totalImagesPullingTime="2.522265825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:33.313124678 +0000 UTC m=+333.438231968" watchObservedRunningTime="2026-10-31 21:21:33.32696609 +0000 UTC m=+333.452073320" Oct 31 21:21:34.501881 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.501851 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h"] Oct 31 21:21:34.511478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.511455 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l"] Oct 31 21:21:34.511628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.511586 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" Oct 31 21:21:34.514486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.514462 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.529195 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.529172 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h"] Oct 31 21:21:34.530771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.530749 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l"] Oct 31 21:21:34.539560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.539517 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 31 21:21:34.539828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.539811 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-mts6t\"" Oct 31 21:21:34.539991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.539975 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 31 21:21:34.539991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.539985 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kdzsw\"" Oct 31 21:21:34.628656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.628627 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26kgm\" (UniqueName: \"kubernetes.io/projected/90d377a0-ae28-4f98-9020-54e37535c7e4-kube-api-access-26kgm\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.628828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.628664 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/90d377a0-ae28-4f98-9020-54e37535c7e4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.628828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.628683 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90d377a0-ae28-4f98-9020-54e37535c7e4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.628907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.628826 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrt2\" (UniqueName: \"kubernetes.io/projected/80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5-kube-api-access-qzrt2\") pod \"limitador-operator-controller-manager-5778bb4449-ltl6h\" (UID: \"80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" Oct 31 21:21:34.730090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.730054 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26kgm\" (UniqueName: \"kubernetes.io/projected/90d377a0-ae28-4f98-9020-54e37535c7e4-kube-api-access-26kgm\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.730090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.730095 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/90d377a0-ae28-4f98-9020-54e37535c7e4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.730291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.730113 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90d377a0-ae28-4f98-9020-54e37535c7e4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.730291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.730143 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrt2\" (UniqueName: \"kubernetes.io/projected/80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5-kube-api-access-qzrt2\") pod \"limitador-operator-controller-manager-5778bb4449-ltl6h\" (UID: \"80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" Oct 31 21:21:34.730839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.730816 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/90d377a0-ae28-4f98-9020-54e37535c7e4-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.732518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.732488 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/90d377a0-ae28-4f98-9020-54e37535c7e4-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.754338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.754276 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26kgm\" (UniqueName: \"kubernetes.io/projected/90d377a0-ae28-4f98-9020-54e37535c7e4-kube-api-access-26kgm\") pod \"kuadrant-console-plugin-66c4fb575d-vpn5l\" (UID: \"90d377a0-ae28-4f98-9020-54e37535c7e4\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:34.755271 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.755249 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrt2\" (UniqueName: \"kubernetes.io/projected/80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5-kube-api-access-qzrt2\") pod \"limitador-operator-controller-manager-5778bb4449-ltl6h\" (UID: \"80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" Oct 31 21:21:34.825050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.825023 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" Oct 31 21:21:34.840250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:34.840231 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" Oct 31 21:21:35.129163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:35.129139 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h"] Oct 31 21:21:35.130124 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:21:35.130099 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e8eb5b_ed9f_4abf_998f_3a92d0a8cca5.slice/crio-b8c24681f022884aaa5d8d3ea965aa6888e25f5e2ad0b98d45eff52bfa2644bc WatchSource:0}: Error finding container b8c24681f022884aaa5d8d3ea965aa6888e25f5e2ad0b98d45eff52bfa2644bc: Status 404 returned error can't find the container with id b8c24681f022884aaa5d8d3ea965aa6888e25f5e2ad0b98d45eff52bfa2644bc Oct 31 21:21:35.140199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:35.140179 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l"] Oct 31 21:21:35.140718 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:21:35.140692 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d377a0_ae28_4f98_9020_54e37535c7e4.slice/crio-dd1e4dc120a3fb6b9baaaaf554b0bab57eb5a271a404758680329d764c29a4a9 WatchSource:0}: Error finding container dd1e4dc120a3fb6b9baaaaf554b0bab57eb5a271a404758680329d764c29a4a9: Status 404 returned error can't find the container with id dd1e4dc120a3fb6b9baaaaf554b0bab57eb5a271a404758680329d764c29a4a9 Oct 31 21:21:35.260429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:35.260394 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" event={"ID":"80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5","Type":"ContainerStarted","Data":"b8c24681f022884aaa5d8d3ea965aa6888e25f5e2ad0b98d45eff52bfa2644bc"} Oct 31 21:21:35.261417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:35.261389 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" event={"ID":"90d377a0-ae28-4f98-9020-54e37535c7e4","Type":"ContainerStarted","Data":"dd1e4dc120a3fb6b9baaaaf554b0bab57eb5a271a404758680329d764c29a4a9"} Oct 31 21:21:39.324860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.324536 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq"] Oct 31 21:21:39.337273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.337243 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.354501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.354468 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq"] Oct 31 21:21:39.364658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.364626 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 21:21:39.364803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.364786 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p69xn\"" Oct 31 21:21:39.365028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.364986 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 21:21:39.474997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.474958 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a2cecb-5a18-4e86-a966-bc536811d816-tls-certs\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.475186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.475025 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c94wr\" (UniqueName: \"kubernetes.io/projected/00a2cecb-5a18-4e86-a966-bc536811d816-kube-api-access-c94wr\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.475186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.475156 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a2cecb-5a18-4e86-a966-bc536811d816-tmp\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.576846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.575666 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a2cecb-5a18-4e86-a966-bc536811d816-tmp\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.576846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.575780 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a2cecb-5a18-4e86-a966-bc536811d816-tls-certs\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.576846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.575814 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c94wr\" (UniqueName: \"kubernetes.io/projected/00a2cecb-5a18-4e86-a966-bc536811d816-kube-api-access-c94wr\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.577687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.577664 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28"] Oct 31 21:21:39.578802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.578744 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/00a2cecb-5a18-4e86-a966-bc536811d816-tmp\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.579406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.579353 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00a2cecb-5a18-4e86-a966-bc536811d816-tls-certs\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.590196 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.590170 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.611548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.611515 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28"] Oct 31 21:21:39.612584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.612556 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c94wr\" (UniqueName: \"kubernetes.io/projected/00a2cecb-5a18-4e86-a966-bc536811d816-kube-api-access-c94wr\") pod \"kube-auth-proxy-5587c8dd5c-gs8xq\" (UID: \"00a2cecb-5a18-4e86-a966-bc536811d816\") " pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.647439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.647411 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-5lr8k\"" Oct 31 21:21:39.662637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.662603 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" Oct 31 21:21:39.676623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676595 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.676739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676639 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.676798 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676749 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.676849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676805 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.676895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676852 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.676942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676900 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.676942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676926 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.677036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676949 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mptkj\" (UniqueName: \"kubernetes.io/projected/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-kube-api-access-mptkj\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.677036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.676975 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778481 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778537 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778576 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778609 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778643 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778690 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778720 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778762 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mptkj\" (UniqueName: \"kubernetes.io/projected/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-kube-api-access-mptkj\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.778972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.778801 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.779711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.779683 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.780021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.780000 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.780275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.780251 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.780555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.780535 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.781819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.781793 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.784813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.784766 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.785484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.785416 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.804631 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.804526 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.808045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.808002 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mptkj\" (UniqueName: \"kubernetes.io/projected/587a84b8-0ce0-4c87-805d-86e2dfea3eb6-kube-api-access-mptkj\") pod \"data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28\" (UID: \"587a84b8-0ce0-4c87-805d-86e2dfea3eb6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:39.933938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:39.932207 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:40.014866 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:40.014777 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq"] Oct 31 21:21:40.291720 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:21:40.291679 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587a84b8_0ce0_4c87_805d_86e2dfea3eb6.slice/crio-7f1707700e6b630555b6789d2bb8e1fdf69896d3e24bc420c3019184db4eea55 WatchSource:0}: Error finding container 7f1707700e6b630555b6789d2bb8e1fdf69896d3e24bc420c3019184db4eea55: Status 404 returned error can't find the container with id 7f1707700e6b630555b6789d2bb8e1fdf69896d3e24bc420c3019184db4eea55 Oct 31 21:21:40.294037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:40.294014 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28"] Oct 31 21:21:40.294489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:40.294441 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:21:40.294605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:40.294516 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:21:40.294605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:40.294550 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:21:40.298169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:40.298136 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" event={"ID":"00a2cecb-5a18-4e86-a966-bc536811d816","Type":"ContainerStarted","Data":"fcf579871885824171ce036f170e2997651fe987f99c3462561328285ad85de9"} Oct 31 21:21:41.310192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.310150 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" event={"ID":"9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777","Type":"ContainerStarted","Data":"f73df9c41893f4f9167bcf0d4c6c53fe6808c4d3ba239ad3852cc82bf5b49404"} Oct 31 21:21:41.311002 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.310978 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" Oct 31 21:21:41.316774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.316739 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" event={"ID":"80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5","Type":"ContainerStarted","Data":"51c099d4bba6ea22965623a616229026d92b8bca27073737770b6e1108d91ecf"} Oct 31 21:21:41.316904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.316811 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" Oct 31 21:21:41.319127 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.319094 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" event={"ID":"587a84b8-0ce0-4c87-805d-86e2dfea3eb6","Type":"ContainerStarted","Data":"8257e45c90f8f4445f3d4dd1ebdafa5194e3f79ebbcaf21f372a8a0b1f08a9d4"} Oct 31 21:21:41.319246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.319149 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" event={"ID":"587a84b8-0ce0-4c87-805d-86e2dfea3eb6","Type":"ContainerStarted","Data":"7f1707700e6b630555b6789d2bb8e1fdf69896d3e24bc420c3019184db4eea55"} Oct 31 21:21:41.405589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.405553 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" podStartSLOduration=1.564839928 podStartE2EDuration="11.405541122s" totalImagesPullingTime="9.840701194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:41.357939798 +0000 UTC m=+341.483047030" watchObservedRunningTime="2026-10-31 21:21:41.405541122 +0000 UTC m=+341.530648352" Oct 31 21:21:41.406179 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.406138 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" podStartSLOduration=1.3742185519999999 podStartE2EDuration="7.406132306s" totalImagesPullingTime="6.031913754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:41.40322503 +0000 UTC m=+341.528332261" watchObservedRunningTime="2026-10-31 21:21:41.406132306 +0000 UTC m=+341.531239541" Oct 31 21:21:41.450729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.450676 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" podStartSLOduration=2.450659086 podStartE2EDuration="2.450659086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:41.449756007 +0000 UTC m=+341.574863239" watchObservedRunningTime="2026-10-31 21:21:41.450659086 +0000 UTC m=+341.575766315" Oct 31 21:21:41.933537 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.933500 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:41.939169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:41.939142 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:42.326817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:42.326767 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" event={"ID":"90d377a0-ae28-4f98-9020-54e37535c7e4","Type":"ContainerStarted","Data":"f21c64428a3413d869c4cae8dd8c179790480ddac5bf27ccb8233ee717ecd368"} Oct 31 21:21:42.328108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:42.328087 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:42.329824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:42.329802 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28" Oct 31 21:21:42.370296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:42.370239 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vpn5l" podStartSLOduration=2.111052019 podStartE2EDuration="8.370223796s" totalImagesPullingTime="6.259171777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:42.368820475 +0000 UTC m=+342.493927707" watchObservedRunningTime="2026-10-31 21:21:42.370223796 +0000 UTC m=+342.495331028" Oct 31 21:21:43.332100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.332052 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" event={"ID":"00a2cecb-5a18-4e86-a966-bc536811d816","Type":"ContainerStarted","Data":"ebe8f42cfba449c2f4d45b58ccf1912656a8970c086b1361d4fc7d4d6d04e74a"} Oct 31 21:21:43.374241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.374193 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5587c8dd5c-gs8xq" podStartSLOduration=1.706515704 podStartE2EDuration="4.374178615s" totalImagesPullingTime="2.667662911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:43.373074745 +0000 UTC m=+343.498181979" watchObservedRunningTime="2026-10-31 21:21:43.374178615 +0000 UTC m=+343.499285846" Oct 31 21:21:43.675628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.675552 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gx9tg"] Oct 31 21:21:43.678676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.678653 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" Oct 31 21:21:43.695251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.695225 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gx9tg"] Oct 31 21:21:43.730262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.730233 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-956cc\"" Oct 31 21:21:43.824626 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.824593 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9bg7\" (UniqueName: \"kubernetes.io/projected/997bdd8d-1915-4267-86ed-2c1f3554d439-kube-api-access-v9bg7\") pod \"authorino-56c6c9ddd5-gx9tg\" (UID: \"997bdd8d-1915-4267-86ed-2c1f3554d439\") " pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" Oct 31 21:21:43.926086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.926014 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9bg7\" (UniqueName: \"kubernetes.io/projected/997bdd8d-1915-4267-86ed-2c1f3554d439-kube-api-access-v9bg7\") pod \"authorino-56c6c9ddd5-gx9tg\" (UID: \"997bdd8d-1915-4267-86ed-2c1f3554d439\") " pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" Oct 31 21:21:43.949275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.949246 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9bg7\" (UniqueName: \"kubernetes.io/projected/997bdd8d-1915-4267-86ed-2c1f3554d439-kube-api-access-v9bg7\") pod \"authorino-56c6c9ddd5-gx9tg\" (UID: \"997bdd8d-1915-4267-86ed-2c1f3554d439\") " pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" Oct 31 21:21:43.990124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:43.990097 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" Oct 31 21:21:44.279293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:44.279260 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gx9tg"] Oct 31 21:21:44.280276 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:21:44.280247 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod997bdd8d_1915_4267_86ed_2c1f3554d439.slice/crio-13103c06c8085474e67b1875ff8f3784a44dc148838b3996e88f580de04a13fb WatchSource:0}: Error finding container 13103c06c8085474e67b1875ff8f3784a44dc148838b3996e88f580de04a13fb: Status 404 returned error can't find the container with id 13103c06c8085474e67b1875ff8f3784a44dc148838b3996e88f580de04a13fb Oct 31 21:21:44.337925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:44.337882 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" event={"ID":"997bdd8d-1915-4267-86ed-2c1f3554d439","Type":"ContainerStarted","Data":"13103c06c8085474e67b1875ff8f3784a44dc148838b3996e88f580de04a13fb"} Oct 31 21:21:46.348658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:46.348618 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" event={"ID":"997bdd8d-1915-4267-86ed-2c1f3554d439","Type":"ContainerStarted","Data":"ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc"} Oct 31 21:21:46.411441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:46.411317 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" podStartSLOduration=1.5352064300000001 podStartE2EDuration="3.411284106s" totalImagesPullingTime="1.876077676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:21:46.4034639 +0000 UTC m=+346.528571132" watchObservedRunningTime="2026-10-31 21:21:46.411284106 +0000 UTC m=+346.536391339" Oct 31 21:21:52.329552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:52.329521 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-ltl6h" Oct 31 21:21:53.334857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:21:53.334827 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-774gh" Oct 31 21:22:01.309801 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:22:01.309765 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:22:01.521288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.521251 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc"] Oct 31 21:22:01.527737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.527714 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:01.542378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.542346 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc"] Oct 31 21:22:01.553079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.553052 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 21:22:01.553211 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.553060 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 21:22:01.554594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.554574 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 21:22:01.554674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.554575 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5vfrq\"" Oct 31 21:22:01.683182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.683094 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed38f675-4a9b-440f-8be4-c617ba27fc1d-data\") pod \"postgres-7cdbf6dc44-wj5cc\" (UID: \"ed38f675-4a9b-440f-8be4-c617ba27fc1d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:01.683353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.683220 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4wl\" (UniqueName: \"kubernetes.io/projected/ed38f675-4a9b-440f-8be4-c617ba27fc1d-kube-api-access-wn4wl\") pod \"postgres-7cdbf6dc44-wj5cc\" (UID: \"ed38f675-4a9b-440f-8be4-c617ba27fc1d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:01.784457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.784415 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4wl\" (UniqueName: \"kubernetes.io/projected/ed38f675-4a9b-440f-8be4-c617ba27fc1d-kube-api-access-wn4wl\") pod \"postgres-7cdbf6dc44-wj5cc\" (UID: \"ed38f675-4a9b-440f-8be4-c617ba27fc1d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:01.784643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.784490 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed38f675-4a9b-440f-8be4-c617ba27fc1d-data\") pod \"postgres-7cdbf6dc44-wj5cc\" (UID: \"ed38f675-4a9b-440f-8be4-c617ba27fc1d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:01.784843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.784823 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ed38f675-4a9b-440f-8be4-c617ba27fc1d-data\") pod \"postgres-7cdbf6dc44-wj5cc\" (UID: \"ed38f675-4a9b-440f-8be4-c617ba27fc1d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:01.808161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.808131 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4wl\" (UniqueName: \"kubernetes.io/projected/ed38f675-4a9b-440f-8be4-c617ba27fc1d-kube-api-access-wn4wl\") pod \"postgres-7cdbf6dc44-wj5cc\" (UID: \"ed38f675-4a9b-440f-8be4-c617ba27fc1d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:01.838552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:01.838521 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:02.129006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:02.128985 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc"] Oct 31 21:22:02.130735 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:22:02.130708 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded38f675_4a9b_440f_8be4_c617ba27fc1d.slice/crio-71afe0ec9c1872ac0c34c681aa38e351ffa2bfb14ec6c3a6a86cfc80c38d6fae WatchSource:0}: Error finding container 71afe0ec9c1872ac0c34c681aa38e351ffa2bfb14ec6c3a6a86cfc80c38d6fae: Status 404 returned error can't find the container with id 71afe0ec9c1872ac0c34c681aa38e351ffa2bfb14ec6c3a6a86cfc80c38d6fae Oct 31 21:22:02.424685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:02.424600 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" event={"ID":"ed38f675-4a9b-440f-8be4-c617ba27fc1d","Type":"ContainerStarted","Data":"71afe0ec9c1872ac0c34c681aa38e351ffa2bfb14ec6c3a6a86cfc80c38d6fae"} Oct 31 21:22:07.450855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:07.450811 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" event={"ID":"ed38f675-4a9b-440f-8be4-c617ba27fc1d","Type":"ContainerStarted","Data":"354ddfdee89df9b43cdaf12360e8eb1e5d8c4b13cb4a8f828eed252134537bf7"} Oct 31 21:22:07.451332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:07.450949 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:07.493774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:07.493724 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" podStartSLOduration=1.818602195 podStartE2EDuration="6.493711604s" totalImagesPullingTime="4.675109409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:07.492330234 +0000 UTC m=+367.617437462" watchObservedRunningTime="2026-10-31 21:22:07.493711604 +0000 UTC m=+367.618818834" Oct 31 21:22:13.484699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:13.484665 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-wj5cc" Oct 31 21:22:31.308902 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:22:31.308862 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:22:47.335557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.335477 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l"] Oct 31 21:22:47.339173 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.339151 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.354761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.354726 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 21:22:47.354892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.354774 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 21:22:47.369340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.369293 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l"] Oct 31 21:22:47.401372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.401343 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 31 21:22:47.401558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.401504 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pc4nc\"" Oct 31 21:22:47.490583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.490548 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w4h5l\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.490583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.490585 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w8xg\" (UniqueName: \"kubernetes.io/projected/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-kube-api-access-4w8xg\") pod \"llmisvc-controller-manager-869d5dcc5d-w4h5l\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.591435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.591356 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w4h5l\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.591435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.591394 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w8xg\" (UniqueName: \"kubernetes.io/projected/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-kube-api-access-4w8xg\") pod \"llmisvc-controller-manager-869d5dcc5d-w4h5l\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.593691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.593670 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w4h5l\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.618055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.618019 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w8xg\" (UniqueName: \"kubernetes.io/projected/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-kube-api-access-4w8xg\") pod \"llmisvc-controller-manager-869d5dcc5d-w4h5l\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.650929 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.650893 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:47.951048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:47.951019 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l"] Oct 31 21:22:47.952414 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:22:47.952383 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod431688d6_dd4b_4a3f_a2c5_6481b8fbb4d4.slice/crio-31e03d56eb459b4032681bb1742ca45f0c08c420de37badf6e6d7181b72dda2d WatchSource:0}: Error finding container 31e03d56eb459b4032681bb1742ca45f0c08c420de37badf6e6d7181b72dda2d: Status 404 returned error can't find the container with id 31e03d56eb459b4032681bb1742ca45f0c08c420de37badf6e6d7181b72dda2d Oct 31 21:22:48.619557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:48.619512 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" event={"ID":"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4","Type":"ContainerStarted","Data":"31e03d56eb459b4032681bb1742ca45f0c08c420de37badf6e6d7181b72dda2d"} Oct 31 21:22:50.630348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:50.630292 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" event={"ID":"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4","Type":"ContainerStarted","Data":"8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09"} Oct 31 21:22:50.630726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:50.630473 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:22:50.674417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:50.674354 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" podStartSLOduration=1.861555337 podStartE2EDuration="3.67433621s" totalImagesPullingTime="1.812780873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:50.672349547 +0000 UTC m=+410.797456779" watchObservedRunningTime="2026-10-31 21:22:50.67433621 +0000 UTC m=+410.799443441" Oct 31 21:22:53.915855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.915822 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp"] Oct 31 21:22:53.919543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.919526 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:53.949332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.949290 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 21:22:53.949453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.949384 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 21:22:53.952000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.951983 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-maas-api-tls\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:53.952050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.952021 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-metrics-certs\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:53.952094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.952057 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rx8x\" (UniqueName: \"kubernetes.io/projected/d473c32e-2a0a-421c-bcd8-ce002cf13a56-kube-api-access-5rx8x\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:53.952094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.952083 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kmnpr\"" Oct 31 21:22:53.952243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.952218 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d473c32e-2a0a-421c-bcd8-ce002cf13a56-openshift-service-ca\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:53.953612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:53.953594 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp"] Oct 31 21:22:54.052654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.052618 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rx8x\" (UniqueName: \"kubernetes.io/projected/d473c32e-2a0a-421c-bcd8-ce002cf13a56-kube-api-access-5rx8x\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.052834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.052675 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d473c32e-2a0a-421c-bcd8-ce002cf13a56-openshift-service-ca\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.052834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.052708 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-maas-api-tls\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.052834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.052735 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-metrics-certs\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.053360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.053331 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d473c32e-2a0a-421c-bcd8-ce002cf13a56-openshift-service-ca\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.055144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.055123 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-maas-api-tls\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.055256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.055233 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-metrics-certs\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.081064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.081030 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rx8x\" (UniqueName: \"kubernetes.io/projected/d473c32e-2a0a-421c-bcd8-ce002cf13a56-kube-api-access-5rx8x\") pod \"maas-api-78f8fc474c-hl9rp\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.231615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.231520 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:54.532185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.532158 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp"] Oct 31 21:22:54.533941 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:22:54.533910 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd473c32e_2a0a_421c_bcd8_ce002cf13a56.slice/crio-7149c87cb75da3858fa0e3583b80b8ba9ab5cd07971513ad3471e9bcefbc3041 WatchSource:0}: Error finding container 7149c87cb75da3858fa0e3583b80b8ba9ab5cd07971513ad3471e9bcefbc3041: Status 404 returned error can't find the container with id 7149c87cb75da3858fa0e3583b80b8ba9ab5cd07971513ad3471e9bcefbc3041 Oct 31 21:22:54.648460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:54.648427 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" event={"ID":"d473c32e-2a0a-421c-bcd8-ce002cf13a56","Type":"ContainerStarted","Data":"7149c87cb75da3858fa0e3583b80b8ba9ab5cd07971513ad3471e9bcefbc3041"} Oct 31 21:22:57.664409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:57.664369 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" event={"ID":"d473c32e-2a0a-421c-bcd8-ce002cf13a56","Type":"ContainerStarted","Data":"06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed"} Oct 31 21:22:57.664818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:57.664550 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:22:57.717880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:22:57.717830 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" podStartSLOduration=1.73571446 podStartE2EDuration="4.717816901s" totalImagesPullingTime="2.982102441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:22:57.716980368 +0000 UTC m=+417.842087597" watchObservedRunningTime="2026-10-31 21:22:57.717816901 +0000 UTC m=+417.842924131" Oct 31 21:23:01.310352 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:23:01.310288 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:23:03.673693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:03.673668 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:23:03.690749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:03.690727 4294 generic.go:356] "Generic (PLEG): container finished" podID="273e8c40-6b3c-4c53-8476-e527511c7340" containerID="af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741" exitCode=0 Oct 31 21:23:03.690846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:03.690794 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" event={"ID":"273e8c40-6b3c-4c53-8476-e527511c7340","Type":"ContainerDied","Data":"af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741"} Oct 31 21:23:03.691090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:03.691077 4294 scope.go:122] "RemoveContainer" containerID="af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741" Oct 31 21:23:03.692025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:03.691999 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:23:03.692100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:03.692063 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:23:03.692100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:03.692096 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:23:04.711295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.711260 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6cv8p"] Oct 31 21:23:04.714974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.714943 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" event={"ID":"273e8c40-6b3c-4c53-8476-e527511c7340","Type":"ContainerStarted","Data":"b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b"} Oct 31 21:23:04.715114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.715053 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:04.727066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.727029 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6cv8p"] Oct 31 21:23:04.802675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.802640 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 21:23:04.802932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.802805 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f23bf6-217d-47a1-be5b-23cf79a33d4b-tls-cert\") pod \"authorino-6889cbd8c4-6cv8p\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:04.803080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.803055 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrxb\" (UniqueName: \"kubernetes.io/projected/00f23bf6-217d-47a1-be5b-23cf79a33d4b-kube-api-access-qsrxb\") pod \"authorino-6889cbd8c4-6cv8p\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:04.898164 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.898129 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6cv8p"] Oct 31 21:23:04.898445 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:23:04.898420 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-qsrxb tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" podUID="00f23bf6-217d-47a1-be5b-23cf79a33d4b" Oct 31 21:23:04.903925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.903897 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f23bf6-217d-47a1-be5b-23cf79a33d4b-tls-cert\") pod \"authorino-6889cbd8c4-6cv8p\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:04.904024 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.903960 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrxb\" (UniqueName: \"kubernetes.io/projected/00f23bf6-217d-47a1-be5b-23cf79a33d4b-kube-api-access-qsrxb\") pod \"authorino-6889cbd8c4-6cv8p\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:04.906268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.906247 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f23bf6-217d-47a1-be5b-23cf79a33d4b-tls-cert\") pod \"authorino-6889cbd8c4-6cv8p\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:04.930415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:04.930395 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrxb\" (UniqueName: \"kubernetes.io/projected/00f23bf6-217d-47a1-be5b-23cf79a33d4b-kube-api-access-qsrxb\") pod \"authorino-6889cbd8c4-6cv8p\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:05.352602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.352567 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:23:05.353900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.353870 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Oct 31 21:23:05.354026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.353922 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Oct 31 21:23:05.715928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.715850 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:05.721950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.721928 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:05.811102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.811071 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f23bf6-217d-47a1-be5b-23cf79a33d4b-kube-api-access-qsrxb\" (UniqueName: \"kubernetes.io/projected/00f23bf6-217d-47a1-be5b-23cf79a33d4b-kube-api-access-qsrxb\") pod \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " Oct 31 21:23:05.811254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.811123 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f23bf6-217d-47a1-be5b-23cf79a33d4b-tls-cert\" (UniqueName: \"kubernetes.io/secret/00f23bf6-217d-47a1-be5b-23cf79a33d4b-tls-cert\") pod \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\" (UID: \"00f23bf6-217d-47a1-be5b-23cf79a33d4b\") " Oct 31 21:23:05.813176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.813150 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f23bf6-217d-47a1-be5b-23cf79a33d4b-kube-api-access-qsrxb" pod "00f23bf6-217d-47a1-be5b-23cf79a33d4b" (UID: "00f23bf6-217d-47a1-be5b-23cf79a33d4b"). InnerVolumeSpecName "kube-api-access-qsrxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:23:05.813279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.813210 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f23bf6-217d-47a1-be5b-23cf79a33d4b-tls-cert" pod "00f23bf6-217d-47a1-be5b-23cf79a33d4b" (UID: "00f23bf6-217d-47a1-be5b-23cf79a33d4b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:23:05.912619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.912577 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsrxb\" (UniqueName: \"kubernetes.io/projected/00f23bf6-217d-47a1-be5b-23cf79a33d4b-kube-api-access-qsrxb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:23:05.912619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:05.912618 4294 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00f23bf6-217d-47a1-be5b-23cf79a33d4b-tls-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:23:06.353001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:06.352920 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Oct 31 21:23:06.353001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:06.352979 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Oct 31 21:23:06.720521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:06.720444 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-6cv8p" Oct 31 21:23:06.800993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:06.800846 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6cv8p"] Oct 31 21:23:06.802403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:06.802373 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-6cv8p"] Oct 31 21:23:06.924001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:06.923967 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gx9tg"] Oct 31 21:23:06.924192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:06.924160 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" podUID="997bdd8d-1915-4267-86ed-2c1f3554d439" containerName="authorino" containerID="cri-o://ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc" gracePeriod=30 Oct 31 21:23:07.227339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.225611 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" Oct 31 21:23:07.324617 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.324529 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997bdd8d-1915-4267-86ed-2c1f3554d439-kube-api-access-v9bg7\" (UniqueName: \"kubernetes.io/projected/997bdd8d-1915-4267-86ed-2c1f3554d439-kube-api-access-v9bg7\") pod \"997bdd8d-1915-4267-86ed-2c1f3554d439\" (UID: \"997bdd8d-1915-4267-86ed-2c1f3554d439\") " Oct 31 21:23:07.326656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.326628 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997bdd8d-1915-4267-86ed-2c1f3554d439-kube-api-access-v9bg7" pod "997bdd8d-1915-4267-86ed-2c1f3554d439" (UID: "997bdd8d-1915-4267-86ed-2c1f3554d439"). InnerVolumeSpecName "kube-api-access-v9bg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:23:07.352968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.352942 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Oct 31 21:23:07.353058 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.352999 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Oct 31 21:23:07.425275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.425242 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9bg7\" (UniqueName: \"kubernetes.io/projected/997bdd8d-1915-4267-86ed-2c1f3554d439-kube-api-access-v9bg7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:23:07.727242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.727158 4294 generic.go:356] "Generic (PLEG): container finished" podID="997bdd8d-1915-4267-86ed-2c1f3554d439" containerID="ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc" exitCode=0 Oct 31 21:23:07.727242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.727200 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" Oct 31 21:23:07.727766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.727241 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" event={"ID":"997bdd8d-1915-4267-86ed-2c1f3554d439","Type":"ContainerDied","Data":"ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc"} Oct 31 21:23:07.727766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.727285 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-gx9tg" event={"ID":"997bdd8d-1915-4267-86ed-2c1f3554d439","Type":"ContainerDied","Data":"13103c06c8085474e67b1875ff8f3784a44dc148838b3996e88f580de04a13fb"} Oct 31 21:23:07.727766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.727322 4294 scope.go:122] "RemoveContainer" containerID="ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc" Oct 31 21:23:07.735465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.735447 4294 scope.go:122] "RemoveContainer" containerID="ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc" Oct 31 21:23:07.735723 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:23:07.735705 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc\": container with ID starting with ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc not found: ID does not exist" containerID="ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc" Oct 31 21:23:07.735768 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.735732 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc"} err="failed to get container status \"ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc\": rpc error: code = NotFound desc = could not find container \"ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc\": container with ID starting with ac58f8202835c221e59c2ef6f4b3528e02a57fa546dcf8aa2cd95ef963ac96dc not found: ID does not exist" Oct 31 21:23:07.796367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.783901 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gx9tg"] Oct 31 21:23:07.799255 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:07.799221 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-gx9tg"] Oct 31 21:23:08.000119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:08.000086 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f23bf6-217d-47a1-be5b-23cf79a33d4b" path="/var/lib/kubelet/pods/00f23bf6-217d-47a1-be5b-23cf79a33d4b/volumes" Oct 31 21:23:08.000338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:08.000325 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997bdd8d-1915-4267-86ed-2c1f3554d439" path="/var/lib/kubelet/pods/997bdd8d-1915-4267-86ed-2c1f3554d439/volumes" Oct 31 21:23:08.353072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:08.353002 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Oct 31 21:23:08.353072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:08.353048 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Oct 31 21:23:09.047469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:09.047428 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:23:09.352978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:09.352892 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" start-of-body= Oct 31 21:23:09.352978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:09.352949 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": dial tcp 10.134.0.16:15021: connect: connection refused" Oct 31 21:23:10.355959 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:10.355933 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:23:10.356864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:10.356841 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:23:14.079103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.079064 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp"] Oct 31 21:23:14.079531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.079349 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" podUID="d473c32e-2a0a-421c-bcd8-ce002cf13a56" containerName="maas-api" containerID="cri-o://06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed" gracePeriod=30 Oct 31 21:23:14.337957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.337897 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:23:14.488725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.488692 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d473c32e-2a0a-421c-bcd8-ce002cf13a56-kube-api-access-5rx8x\" (UniqueName: \"kubernetes.io/projected/d473c32e-2a0a-421c-bcd8-ce002cf13a56-kube-api-access-5rx8x\") pod \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " Oct 31 21:23:14.488912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.488743 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d473c32e-2a0a-421c-bcd8-ce002cf13a56-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d473c32e-2a0a-421c-bcd8-ce002cf13a56-openshift-service-ca\") pod \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " Oct 31 21:23:14.488912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.488780 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-maas-api-tls\") pod \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " Oct 31 21:23:14.488912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.488855 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-metrics-certs\") pod \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\" (UID: \"d473c32e-2a0a-421c-bcd8-ce002cf13a56\") " Oct 31 21:23:14.489184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.489160 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d473c32e-2a0a-421c-bcd8-ce002cf13a56-openshift-service-ca" pod "d473c32e-2a0a-421c-bcd8-ce002cf13a56" (UID: "d473c32e-2a0a-421c-bcd8-ce002cf13a56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:23:14.491038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.491008 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d473c32e-2a0a-421c-bcd8-ce002cf13a56-kube-api-access-5rx8x" pod "d473c32e-2a0a-421c-bcd8-ce002cf13a56" (UID: "d473c32e-2a0a-421c-bcd8-ce002cf13a56"). InnerVolumeSpecName "kube-api-access-5rx8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:23:14.491038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.491025 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-metrics-certs" pod "d473c32e-2a0a-421c-bcd8-ce002cf13a56" (UID: "d473c32e-2a0a-421c-bcd8-ce002cf13a56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:23:14.491182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.491044 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-maas-api-tls" pod "d473c32e-2a0a-421c-bcd8-ce002cf13a56" (UID: "d473c32e-2a0a-421c-bcd8-ce002cf13a56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:23:14.589767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.589684 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rx8x\" (UniqueName: \"kubernetes.io/projected/d473c32e-2a0a-421c-bcd8-ce002cf13a56-kube-api-access-5rx8x\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:23:14.589767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.589721 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d473c32e-2a0a-421c-bcd8-ce002cf13a56-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:23:14.589767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.589733 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:23:14.589767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.589741 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d473c32e-2a0a-421c-bcd8-ce002cf13a56-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:23:14.759611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.759579 4294 generic.go:356] "Generic (PLEG): container finished" podID="d473c32e-2a0a-421c-bcd8-ce002cf13a56" containerID="06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed" exitCode=0 Oct 31 21:23:14.759777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.759630 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" Oct 31 21:23:14.759777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.759659 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" event={"ID":"d473c32e-2a0a-421c-bcd8-ce002cf13a56","Type":"ContainerDied","Data":"06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed"} Oct 31 21:23:14.759777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.759712 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp" event={"ID":"d473c32e-2a0a-421c-bcd8-ce002cf13a56","Type":"ContainerDied","Data":"7149c87cb75da3858fa0e3583b80b8ba9ab5cd07971513ad3471e9bcefbc3041"} Oct 31 21:23:14.759777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.759733 4294 scope.go:122] "RemoveContainer" containerID="06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed" Oct 31 21:23:14.769055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.769040 4294 scope.go:122] "RemoveContainer" containerID="06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed" Oct 31 21:23:14.769328 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:23:14.769291 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed\": container with ID starting with 06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed not found: ID does not exist" containerID="06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed" Oct 31 21:23:14.769404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.769337 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed"} err="failed to get container status \"06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed\": rpc error: code = NotFound desc = could not find container \"06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed\": container with ID starting with 06fc2f0f3bd115b040931e0f1c862e7f40a491919f73cf42fac29fc1743ed2ed not found: ID does not exist" Oct 31 21:23:14.812366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.812334 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp"] Oct 31 21:23:14.823877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:14.823852 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78f8fc474c-hl9rp"] Oct 31 21:23:15.982412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:15.982377 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d473c32e-2a0a-421c-bcd8-ce002cf13a56" path="/var/lib/kubelet/pods/d473c32e-2a0a-421c-bcd8-ce002cf13a56/volumes" Oct 31 21:23:21.636508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:21.636471 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:23:31.309600 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:23:31.309560 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:23:48.235524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.235493 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl"] Oct 31 21:23:48.235894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.235815 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d473c32e-2a0a-421c-bcd8-ce002cf13a56" containerName="maas-api" Oct 31 21:23:48.235894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.235826 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d473c32e-2a0a-421c-bcd8-ce002cf13a56" containerName="maas-api" Oct 31 21:23:48.235894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.235835 4294 cpu_manager.go:414] "containerMap: removing container" podUID="997bdd8d-1915-4267-86ed-2c1f3554d439" containerName="authorino" Oct 31 21:23:48.235894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.235839 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997bdd8d-1915-4267-86ed-2c1f3554d439" containerName="authorino" Oct 31 21:23:48.236014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.235922 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d473c32e-2a0a-421c-bcd8-ce002cf13a56" containerName="maas-api" Oct 31 21:23:48.236014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.235928 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="997bdd8d-1915-4267-86ed-2c1f3554d439" containerName="authorino" Oct 31 21:23:48.237722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.237705 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.261486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.261460 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl"] Oct 31 21:23:48.415497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.415451 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26251b9d-2632-449f-9e30-eecdf1204060-cert\") pod \"llmisvc-controller-manager-6466c554d8-7m5zl\" (UID: \"26251b9d-2632-449f-9e30-eecdf1204060\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.415658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.415526 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qmv\" (UniqueName: \"kubernetes.io/projected/26251b9d-2632-449f-9e30-eecdf1204060-kube-api-access-l8qmv\") pod \"llmisvc-controller-manager-6466c554d8-7m5zl\" (UID: \"26251b9d-2632-449f-9e30-eecdf1204060\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.516565 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.516480 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26251b9d-2632-449f-9e30-eecdf1204060-cert\") pod \"llmisvc-controller-manager-6466c554d8-7m5zl\" (UID: \"26251b9d-2632-449f-9e30-eecdf1204060\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.516565 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.516539 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qmv\" (UniqueName: \"kubernetes.io/projected/26251b9d-2632-449f-9e30-eecdf1204060-kube-api-access-l8qmv\") pod \"llmisvc-controller-manager-6466c554d8-7m5zl\" (UID: \"26251b9d-2632-449f-9e30-eecdf1204060\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.518721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.518694 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26251b9d-2632-449f-9e30-eecdf1204060-cert\") pod \"llmisvc-controller-manager-6466c554d8-7m5zl\" (UID: \"26251b9d-2632-449f-9e30-eecdf1204060\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.540710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.540681 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qmv\" (UniqueName: \"kubernetes.io/projected/26251b9d-2632-449f-9e30-eecdf1204060-kube-api-access-l8qmv\") pod \"llmisvc-controller-manager-6466c554d8-7m5zl\" (UID: \"26251b9d-2632-449f-9e30-eecdf1204060\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.548521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.548504 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:48.843523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.843499 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl"] Oct 31 21:23:48.844332 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:23:48.844284 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26251b9d_2632_449f_9e30_eecdf1204060.slice/crio-4bf376c96c9b0914849062ce36304fd83c9a91203b7e8bcaf21dcbb27ef702b1 WatchSource:0}: Error finding container 4bf376c96c9b0914849062ce36304fd83c9a91203b7e8bcaf21dcbb27ef702b1: Status 404 returned error can't find the container with id 4bf376c96c9b0914849062ce36304fd83c9a91203b7e8bcaf21dcbb27ef702b1 Oct 31 21:23:48.902147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:48.902112 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" event={"ID":"26251b9d-2632-449f-9e30-eecdf1204060","Type":"ContainerStarted","Data":"4bf376c96c9b0914849062ce36304fd83c9a91203b7e8bcaf21dcbb27ef702b1"} Oct 31 21:23:49.911069 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:49.911035 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" event={"ID":"26251b9d-2632-449f-9e30-eecdf1204060","Type":"ContainerStarted","Data":"91518ef5debabc97bb495f27a5901c84d943aff3fe538eaca9032cd12e5eac34"} Oct 31 21:23:49.911476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:49.911176 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:23:49.953141 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:23:49.953099 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" podStartSLOduration=1.699010877 podStartE2EDuration="1.95308739s" totalImagesPullingTime="254.076513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:23:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:23:49.951684007 +0000 UTC m=+470.076791237" watchObservedRunningTime="2026-10-31 21:23:49.95308739 +0000 UTC m=+470.078194660" Oct 31 21:24:01.309034 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:24:01.308997 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:24:20.917906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:20.917831 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-7m5zl" Oct 31 21:24:20.991364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:20.991331 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l"] Oct 31 21:24:20.991572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:20.991536 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" podUID="431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" containerName="manager" containerID="cri-o://8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09" gracePeriod=30 Oct 31 21:24:21.232907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:21.232879 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:24:21.253026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:21.252996 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-kube-api-access-4w8xg\" (UniqueName: \"kubernetes.io/projected/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-kube-api-access-4w8xg\") pod \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " Oct 31 21:24:21.253159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:21.253030 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-cert\" (UniqueName: \"kubernetes.io/secret/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-cert\") pod \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\" (UID: \"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4\") " Oct 31 21:24:21.255103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:21.255075 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-kube-api-access-4w8xg" pod "431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" (UID: "431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4"). InnerVolumeSpecName "kube-api-access-4w8xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:24:21.255212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:21.255097 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-cert" pod "431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" (UID: "431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:24:21.354270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:21.354231 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w8xg\" (UniqueName: \"kubernetes.io/projected/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-kube-api-access-4w8xg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:24:21.354270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:21.354264 4294 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:24:22.051610 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.051576 4294 generic.go:356] "Generic (PLEG): container finished" podID="431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" containerID="8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09" exitCode=0 Oct 31 21:24:22.052026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.051637 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" Oct 31 21:24:22.052026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.051657 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" event={"ID":"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4","Type":"ContainerDied","Data":"8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09"} Oct 31 21:24:22.052026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.051720 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l" event={"ID":"431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4","Type":"ContainerDied","Data":"31e03d56eb459b4032681bb1742ca45f0c08c420de37badf6e6d7181b72dda2d"} Oct 31 21:24:22.052026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.051743 4294 scope.go:122] "RemoveContainer" containerID="8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09" Oct 31 21:24:22.060342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.060325 4294 scope.go:122] "RemoveContainer" containerID="8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09" Oct 31 21:24:22.060632 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:24:22.060614 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09\": container with ID starting with 8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09 not found: ID does not exist" containerID="8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09" Oct 31 21:24:22.060702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.060642 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09"} err="failed to get container status \"8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09\": rpc error: code = NotFound desc = could not find container \"8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09\": container with ID starting with 8f2b8fdd3fce4dbe603d9b7c6621281d5afe04d1c45adc9a66524834239e2a09 not found: ID does not exist" Oct 31 21:24:22.103643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.103615 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l"] Oct 31 21:24:22.115646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:22.115620 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w4h5l"] Oct 31 21:24:23.984178 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:24:23.984142 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" path="/var/lib/kubelet/pods/431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4/volumes" Oct 31 21:24:31.309955 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:24:31.309913 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:25:01.308710 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:25:01.308676 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:25:31.309178 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:25:31.309137 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:25:44.029583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.029509 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c695f6bc5-nbdsf"] Oct 31 21:25:44.030025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.029870 4294 cpu_manager.go:414] "containerMap: removing container" podUID="431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" containerName="manager" Oct 31 21:25:44.030025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.029882 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" containerName="manager" Oct 31 21:25:44.030025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.029931 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="431688d6-dd4b-4a3f-a2c5-6481b8fbb4d4" containerName="manager" Oct 31 21:25:44.033079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.033060 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.050280 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.050253 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c695f6bc5-nbdsf"] Oct 31 21:25:44.060899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.060874 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 21:25:44.060997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.060952 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-956cc\"" Oct 31 21:25:44.149604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.149555 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbeb690d-72d7-44bc-bc54-e4512acd0d09-tls-cert\") pod \"authorino-5c695f6bc5-nbdsf\" (UID: \"dbeb690d-72d7-44bc-bc54-e4512acd0d09\") " pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.149776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.149724 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpnf4\" (UniqueName: \"kubernetes.io/projected/dbeb690d-72d7-44bc-bc54-e4512acd0d09-kube-api-access-qpnf4\") pod \"authorino-5c695f6bc5-nbdsf\" (UID: \"dbeb690d-72d7-44bc-bc54-e4512acd0d09\") " pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.250570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.250536 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpnf4\" (UniqueName: \"kubernetes.io/projected/dbeb690d-72d7-44bc-bc54-e4512acd0d09-kube-api-access-qpnf4\") pod \"authorino-5c695f6bc5-nbdsf\" (UID: \"dbeb690d-72d7-44bc-bc54-e4512acd0d09\") " pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.250706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.250587 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbeb690d-72d7-44bc-bc54-e4512acd0d09-tls-cert\") pod \"authorino-5c695f6bc5-nbdsf\" (UID: \"dbeb690d-72d7-44bc-bc54-e4512acd0d09\") " pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.253005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.252983 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/dbeb690d-72d7-44bc-bc54-e4512acd0d09-tls-cert\") pod \"authorino-5c695f6bc5-nbdsf\" (UID: \"dbeb690d-72d7-44bc-bc54-e4512acd0d09\") " pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.273695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.273664 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpnf4\" (UniqueName: \"kubernetes.io/projected/dbeb690d-72d7-44bc-bc54-e4512acd0d09-kube-api-access-qpnf4\") pod \"authorino-5c695f6bc5-nbdsf\" (UID: \"dbeb690d-72d7-44bc-bc54-e4512acd0d09\") " pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.343900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.343849 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" Oct 31 21:25:44.643563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:44.643536 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c695f6bc5-nbdsf"] Oct 31 21:25:44.644738 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:25:44.644713 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbeb690d_72d7_44bc_bc54_e4512acd0d09.slice/crio-4787ea8e200fff06cb7fcfda97328a5b3e76cb2ad1c6e06e192949cbf3e28760 WatchSource:0}: Error finding container 4787ea8e200fff06cb7fcfda97328a5b3e76cb2ad1c6e06e192949cbf3e28760: Status 404 returned error can't find the container with id 4787ea8e200fff06cb7fcfda97328a5b3e76cb2ad1c6e06e192949cbf3e28760 Oct 31 21:25:45.400216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:45.400186 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" event={"ID":"dbeb690d-72d7-44bc-bc54-e4512acd0d09","Type":"ContainerStarted","Data":"5550d2c374e7ac786753ec03c7e1f8442c40254e82b616a9fd4eec10e4b9a42c"} Oct 31 21:25:45.400580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:45.400232 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" event={"ID":"dbeb690d-72d7-44bc-bc54-e4512acd0d09","Type":"ContainerStarted","Data":"4787ea8e200fff06cb7fcfda97328a5b3e76cb2ad1c6e06e192949cbf3e28760"} Oct 31 21:25:45.440843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:25:45.440793 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5c695f6bc5-nbdsf" podStartSLOduration=0.773342306 podStartE2EDuration="1.440779806s" totalImagesPullingTime="667.4375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:25:45.440425438 +0000 UTC m=+585.565532668" watchObservedRunningTime="2026-10-31 21:25:45.440779806 +0000 UTC m=+585.565887036" Oct 31 21:26:00.007637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:00.007613 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:26:00.008054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:00.007617 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:26:01.308557 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:26:01.308526 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:26:05.016607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.016569 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl"] Oct 31 21:26:05.020243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.020224 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.048845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.048820 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl"] Oct 31 21:26:05.127846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.127809 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.127982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.127859 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb69f48-47a6-492f-9564-33f725285f09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.127982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.127930 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.127982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.127964 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb69f48-47a6-492f-9564-33f725285f09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.128088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.127989 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.128088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.128019 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.128151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.128131 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb69f48-47a6-492f-9564-33f725285f09-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.128183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.128159 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9tdw\" (UniqueName: \"kubernetes.io/projected/ddb69f48-47a6-492f-9564-33f725285f09-kube-api-access-h9tdw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.128215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.128183 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229128 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229100 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb69f48-47a6-492f-9564-33f725285f09-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229132 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9tdw\" (UniqueName: \"kubernetes.io/projected/ddb69f48-47a6-492f-9564-33f725285f09-kube-api-access-h9tdw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229150 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229188 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229210 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb69f48-47a6-492f-9564-33f725285f09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229234 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229276 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb69f48-47a6-492f-9564-33f725285f09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229344 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229378 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229642 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229689 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229759 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.229802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.229793 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.230068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.230045 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb69f48-47a6-492f-9564-33f725285f09-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.231543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.231521 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb69f48-47a6-492f-9564-33f725285f09-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.231709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.231688 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb69f48-47a6-492f-9564-33f725285f09-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.252935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.252907 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb69f48-47a6-492f-9564-33f725285f09-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.253011 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.252968 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9tdw\" (UniqueName: \"kubernetes.io/projected/ddb69f48-47a6-492f-9564-33f725285f09-kube-api-access-h9tdw\") pod \"maas-default-gateway-openshift-default-58847f9fc5-c7qkl\" (UID: \"ddb69f48-47a6-492f-9564-33f725285f09\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.333714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.333665 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:05.645499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.645472 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl"] Oct 31 21:26:05.646274 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:26:05.646244 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb69f48_47a6_492f_9564_33f725285f09.slice/crio-7ec50a4f3032a89fbc7198627231fb4572e76d061c614460a0df5a7607d31a15 WatchSource:0}: Error finding container 7ec50a4f3032a89fbc7198627231fb4572e76d061c614460a0df5a7607d31a15: Status 404 returned error can't find the container with id 7ec50a4f3032a89fbc7198627231fb4572e76d061c614460a0df5a7607d31a15 Oct 31 21:26:05.648438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.648406 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:26:05.648512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.648475 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:26:05.648512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:05.648504 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:26:06.499522 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:06.499477 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" event={"ID":"ddb69f48-47a6-492f-9564-33f725285f09","Type":"ContainerStarted","Data":"41e7af2596ae080245310019e473d5fedb4d22534e822fd9d5d477002a198da7"} Oct 31 21:26:06.500022 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:06.499540 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" event={"ID":"ddb69f48-47a6-492f-9564-33f725285f09","Type":"ContainerStarted","Data":"7ec50a4f3032a89fbc7198627231fb4572e76d061c614460a0df5a7607d31a15"} Oct 31 21:26:06.557104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:06.557055 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podStartSLOduration=2.557040541 podStartE2EDuration="2.557040541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:06.556157895 +0000 UTC m=+606.681265127" watchObservedRunningTime="2026-10-31 21:26:06.557040541 +0000 UTC m=+606.682147771" Oct 31 21:26:07.334376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:07.334342 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:07.335482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:07.335452 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7qkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 21:26:07.335595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:07.335523 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podUID="ddb69f48-47a6-492f-9564-33f725285f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 21:26:08.334978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:08.334943 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7qkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 21:26:08.335388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:08.334996 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podUID="ddb69f48-47a6-492f-9564-33f725285f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 21:26:09.334597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:09.334558 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7qkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 21:26:09.334766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:09.334610 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podUID="ddb69f48-47a6-492f-9564-33f725285f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 21:26:10.334277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:10.334242 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7qkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 21:26:10.334668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:10.334326 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podUID="ddb69f48-47a6-492f-9564-33f725285f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 21:26:11.334100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:11.334057 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7qkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 21:26:11.334278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:11.334126 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podUID="ddb69f48-47a6-492f-9564-33f725285f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 21:26:12.334435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:12.334400 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7qkl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 31 21:26:12.334824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:12.334456 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podUID="ddb69f48-47a6-492f-9564-33f725285f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 31 21:26:13.509496 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:13.509466 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:13.509937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:13.509882 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:13.510701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:13.510680 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" Oct 31 21:26:20.052761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.052730 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq"] Oct 31 21:26:20.056006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.055984 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.080960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.080932 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq"] Oct 31 21:26:20.088144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.088115 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.088269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.088220 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121bcf5-aee6-409b-982a-6f2939da08b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.088368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.088281 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvm8w\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-kube-api-access-pvm8w\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.088707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.088681 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.090388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.089292 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.090388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.089343 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.090388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.089396 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.090388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.089434 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.090388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.089501 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.190678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190646 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.190864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190698 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121bcf5-aee6-409b-982a-6f2939da08b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.190864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190732 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvm8w\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-kube-api-access-pvm8w\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.190864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190773 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.190864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190850 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190882 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190909 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190947 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.190976 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.191055 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.191203 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.191329 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.191351 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.191591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.191574 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121bcf5-aee6-409b-982a-6f2939da08b6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.193318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.193276 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.193559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.193540 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.213639 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.213613 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.213719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.213702 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvm8w\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-kube-api-access-pvm8w\") pod \"maas-default-gateway-openshift-default-58847f9fc5-bncsq\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.369577 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.369500 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:20.680214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.680190 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq"] Oct 31 21:26:20.681587 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:26:20.681565 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0121bcf5_aee6_409b_982a_6f2939da08b6.slice/crio-455eb8f14b596fd7130594dc5eb8748b927bfcee692475b52c1bf4c3746b5730 WatchSource:0}: Error finding container 455eb8f14b596fd7130594dc5eb8748b927bfcee692475b52c1bf4c3746b5730: Status 404 returned error can't find the container with id 455eb8f14b596fd7130594dc5eb8748b927bfcee692475b52c1bf4c3746b5730 Oct 31 21:26:20.683634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.683605 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:26:20.683706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.683674 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:26:20.683706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:20.683699 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:26:21.564994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.564959 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" event={"ID":"0121bcf5-aee6-409b-982a-6f2939da08b6","Type":"ContainerStarted","Data":"4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776"} Oct 31 21:26:21.565376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.565033 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" event={"ID":"0121bcf5-aee6-409b-982a-6f2939da08b6","Type":"ContainerStarted","Data":"455eb8f14b596fd7130594dc5eb8748b927bfcee692475b52c1bf4c3746b5730"} Oct 31 21:26:21.647814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.647761 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podStartSLOduration=1.64774741 podStartE2EDuration="1.64774741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:21.634248823 +0000 UTC m=+621.759356054" watchObservedRunningTime="2026-10-31 21:26:21.64774741 +0000 UTC m=+621.772854705" Oct 31 21:26:21.804539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.804504 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h"] Oct 31 21:26:21.807533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.807515 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:21.822472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.822447 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-353db9f5-serving-cert\"" Oct 31 21:26:21.822472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.822468 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 21:26:21.822632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.822525 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kmnpr\"" Oct 31 21:26:21.827611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.827590 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h"] Oct 31 21:26:21.906612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.906579 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-openshift-service-ca\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:21.906612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.906614 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t79p7\" (UniqueName: \"kubernetes.io/projected/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-kube-api-access-t79p7\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:21.906815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.906716 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-metrics-certs\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:21.906815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:21.906766 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-maas-api-tls\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.008163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.008130 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-maas-api-tls\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.008343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.008193 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-openshift-service-ca\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.008343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.008214 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t79p7\" (UniqueName: \"kubernetes.io/projected/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-kube-api-access-t79p7\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.008343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.008274 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-metrics-certs\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.008938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.008912 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-openshift-service-ca\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.010668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.010642 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-maas-api-tls\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.010761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.010684 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-metrics-certs\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.035690 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.035657 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t79p7\" (UniqueName: \"kubernetes.io/projected/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-kube-api-access-t79p7\") pod \"maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.124180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.124106 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:22.369969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.369931 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:22.371178 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.371157 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 21:26:22.371233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.371203 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 21:26:22.450422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.450398 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h"] Oct 31 21:26:22.452107 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:26:22.452078 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1912e5d_3dc3_4401_b51d_aa9a52e89c10.slice/crio-2768c945b230b62c858b6c4b7a4886c4e0e6140b43fa1550ac34fb8390f5056f WatchSource:0}: Error finding container 2768c945b230b62c858b6c4b7a4886c4e0e6140b43fa1550ac34fb8390f5056f: Status 404 returned error can't find the container with id 2768c945b230b62c858b6c4b7a4886c4e0e6140b43fa1550ac34fb8390f5056f Oct 31 21:26:22.453612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.453593 4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:26:22.597568 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:22.597529 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" event={"ID":"f1912e5d-3dc3-4401-b51d-aa9a52e89c10","Type":"ContainerStarted","Data":"2768c945b230b62c858b6c4b7a4886c4e0e6140b43fa1550ac34fb8390f5056f"} Oct 31 21:26:23.370153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:23.370121 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 21:26:23.370345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:23.370173 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 21:26:23.603756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:23.603720 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" event={"ID":"f1912e5d-3dc3-4401-b51d-aa9a52e89c10","Type":"ContainerStarted","Data":"ed1e2109940f67d3032b390ec39dc2cbcd4cbb91a6a04ad5e1863f13660ebe00"} Oct 31 21:26:23.604155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:23.603820 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:23.645683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:23.645583 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" podStartSLOduration=2.173264749 podStartE2EDuration="2.645569078s" totalImagesPullingTime="472.304329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:23.644618103 +0000 UTC m=+623.769725335" watchObservedRunningTime="2026-10-31 21:26:23.645569078 +0000 UTC m=+623.770676311" Oct 31 21:26:24.370515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:24.370483 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 21:26:24.370701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:24.370542 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 21:26:25.370136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:25.370095 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 21:26:25.370638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:25.370150 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 21:26:26.369867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:26.369827 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 21:26:26.370063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:26.369887 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 21:26:27.370027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:27.369975 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-bncsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 31 21:26:27.370531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:27.370040 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 31 21:26:28.833597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:28.833565 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:28.834062 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:28.833876 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:28.834699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:28.834678 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:26:29.321004 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.320972 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf"] Oct 31 21:26:29.324526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.324509 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.346005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.345967 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf"] Oct 31 21:26:29.352864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.352842 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ac0736b6-serving-cert\"" Oct 31 21:26:29.374678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.374648 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-maas-api-tls\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.374788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.374686 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-openshift-service-ca\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.374858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.374830 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5t5\" (UniqueName: \"kubernetes.io/projected/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-kube-api-access-jd5t5\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.374918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.374874 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-metrics-certs\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.475401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.475368 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-maas-api-tls\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.475401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.475404 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-openshift-service-ca\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.475619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.475433 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5t5\" (UniqueName: \"kubernetes.io/projected/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-kube-api-access-jd5t5\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.475675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.475612 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-metrics-certs\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.476064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.476036 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-openshift-service-ca\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.477877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.477849 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-metrics-certs\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.477980 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.477904 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-maas-api-tls\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.499576 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.499547 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5t5\" (UniqueName: \"kubernetes.io/projected/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-kube-api-access-jd5t5\") pod \"maas-api-e2e-ait-ac0736b6-6848b48848-gskbf\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.614540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.614437 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:29.636927 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.636898 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:29.935017 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:29.934988 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf"] Oct 31 21:26:29.936247 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:26:29.936220 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb8afc2_9081_434d_8a1c_2450a8e47cd4.slice/crio-91665e7f1f72f2fcf34c8f159717fe16f809cef6cd04e5eb3107a556ab49cc2c WatchSource:0}: Error finding container 91665e7f1f72f2fcf34c8f159717fe16f809cef6cd04e5eb3107a556ab49cc2c: Status 404 returned error can't find the container with id 91665e7f1f72f2fcf34c8f159717fe16f809cef6cd04e5eb3107a556ab49cc2c Oct 31 21:26:30.635649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:30.635612 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" event={"ID":"0fb8afc2-9081-434d-8a1c-2450a8e47cd4","Type":"ContainerStarted","Data":"2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556"} Oct 31 21:26:30.635841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:30.635750 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:30.635841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:30.635777 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" event={"ID":"0fb8afc2-9081-434d-8a1c-2450a8e47cd4","Type":"ContainerStarted","Data":"91665e7f1f72f2fcf34c8f159717fe16f809cef6cd04e5eb3107a556ab49cc2c"} Oct 31 21:26:30.679277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:30.679210 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" podStartSLOduration=1.387541894 podStartE2EDuration="1.679191929s" totalImagesPullingTime="291.650035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:30.677475737 +0000 UTC m=+630.802582969" watchObservedRunningTime="2026-10-31 21:26:30.679191929 +0000 UTC m=+630.804299160" Oct 31 21:26:31.311493 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:26:31.311459 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:26:36.646451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:36.646418 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:26:37.002096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.002064 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw"] Oct 31 21:26:37.005541 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.005523 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.020157 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.020137 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-7ef0e96a-serving-cert\"" Oct 31 21:26:37.035365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.035335 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw"] Oct 31 21:26:37.051448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.051415 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gpz\" (UniqueName: \"kubernetes.io/projected/c2834b9c-0d60-442f-8c38-6daace8e9dc5-kube-api-access-k4gpz\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.051612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.051465 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-metrics-certs\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.051612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.051574 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2834b9c-0d60-442f-8c38-6daace8e9dc5-openshift-service-ca\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.051704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.051680 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.153324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.153277 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gpz\" (UniqueName: \"kubernetes.io/projected/c2834b9c-0d60-442f-8c38-6daace8e9dc5-kube-api-access-k4gpz\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.153529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.153340 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-metrics-certs\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.153529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.153368 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2834b9c-0d60-442f-8c38-6daace8e9dc5-openshift-service-ca\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.153529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.153399 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.153702 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:26:37.153530 4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-serving-cert: secret "maas-api-e2e-derive-7ef0e96a-serving-cert" not found Oct 31 21:26:37.153702 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:26:37.153606 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls podName:c2834b9c-0d60-442f-8c38-6daace8e9dc5 nodeName:}" failed. No retries permitted until 2026-10-31 21:26:37.653586019 +0000 UTC m=+637.778693245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls") pod "maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" (UID: "c2834b9c-0d60-442f-8c38-6daace8e9dc5") : secret "maas-api-e2e-derive-7ef0e96a-serving-cert" not found Oct 31 21:26:37.154126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.154102 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2834b9c-0d60-442f-8c38-6daace8e9dc5-openshift-service-ca\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.155739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.155712 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-metrics-certs\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.179207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.179173 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gpz\" (UniqueName: \"kubernetes.io/projected/c2834b9c-0d60-442f-8c38-6daace8e9dc5-kube-api-access-k4gpz\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.657847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.657806 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.660155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.660128 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls\") pod \"maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:37.918629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:37.918542 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:38.219746 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:38.219717 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw"] Oct 31 21:26:38.224368 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:26:38.224332 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2834b9c_0d60_442f_8c38_6daace8e9dc5.slice/crio-78bc580f0d89c1f2d406b8d9a643ebd9b395a9a01fda3a8cea734d786847fdca WatchSource:0}: Error finding container 78bc580f0d89c1f2d406b8d9a643ebd9b395a9a01fda3a8cea734d786847fdca: Status 404 returned error can't find the container with id 78bc580f0d89c1f2d406b8d9a643ebd9b395a9a01fda3a8cea734d786847fdca Oct 31 21:26:38.673108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:38.673074 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" event={"ID":"c2834b9c-0d60-442f-8c38-6daace8e9dc5","Type":"ContainerStarted","Data":"7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91"} Oct 31 21:26:38.673578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:38.673129 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" event={"ID":"c2834b9c-0d60-442f-8c38-6daace8e9dc5","Type":"ContainerStarted","Data":"78bc580f0d89c1f2d406b8d9a643ebd9b395a9a01fda3a8cea734d786847fdca"} Oct 31 21:26:38.673578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:38.673219 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:38.716606 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:38.716560 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" podStartSLOduration=2.407670435 podStartE2EDuration="2.716545958s" totalImagesPullingTime="308.875523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:26:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:26:38.714836487 +0000 UTC m=+638.839943740" watchObservedRunningTime="2026-10-31 21:26:38.716545958 +0000 UTC m=+638.841653186" Oct 31 21:26:44.682441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:44.682413 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:26:54.615383 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:54.615346 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h"] Oct 31 21:26:54.615885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:54.615586 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" podUID="f1912e5d-3dc3-4401-b51d-aa9a52e89c10" containerName="maas-api" containerID="cri-o://ed1e2109940f67d3032b390ec39dc2cbcd4cbb91a6a04ad5e1863f13660ebe00" gracePeriod=30 Oct 31 21:26:54.746458 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:54.746425 4294 generic.go:356] "Generic (PLEG): container finished" podID="f1912e5d-3dc3-4401-b51d-aa9a52e89c10" containerID="ed1e2109940f67d3032b390ec39dc2cbcd4cbb91a6a04ad5e1863f13660ebe00" exitCode=0 Oct 31 21:26:54.746578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:54.746491 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" event={"ID":"f1912e5d-3dc3-4401-b51d-aa9a52e89c10","Type":"ContainerDied","Data":"ed1e2109940f67d3032b390ec39dc2cbcd4cbb91a6a04ad5e1863f13660ebe00"} Oct 31 21:26:54.875078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:54.874994 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:55.026038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.025994 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-openshift-service-ca\") pod \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " Oct 31 21:26:55.026212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.026047 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-metrics-certs\") pod \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " Oct 31 21:26:55.026212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.026099 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-maas-api-tls\") pod \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " Oct 31 21:26:55.026212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.026132 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-kube-api-access-t79p7\" (UniqueName: \"kubernetes.io/projected/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-kube-api-access-t79p7\") pod \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\" (UID: \"f1912e5d-3dc3-4401-b51d-aa9a52e89c10\") " Oct 31 21:26:55.026416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.026394 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-openshift-service-ca" pod "f1912e5d-3dc3-4401-b51d-aa9a52e89c10" (UID: "f1912e5d-3dc3-4401-b51d-aa9a52e89c10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:26:55.026529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.026511 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:26:55.028734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.028698 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-maas-api-tls" pod "f1912e5d-3dc3-4401-b51d-aa9a52e89c10" (UID: "f1912e5d-3dc3-4401-b51d-aa9a52e89c10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:26:55.029119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.029091 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-metrics-certs" pod "f1912e5d-3dc3-4401-b51d-aa9a52e89c10" (UID: "f1912e5d-3dc3-4401-b51d-aa9a52e89c10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:26:55.032489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.032451 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-kube-api-access-t79p7" pod "f1912e5d-3dc3-4401-b51d-aa9a52e89c10" (UID: "f1912e5d-3dc3-4401-b51d-aa9a52e89c10"). InnerVolumeSpecName "kube-api-access-t79p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:26:55.127332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.127236 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:26:55.127332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.127275 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:26:55.127538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.127294 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t79p7\" (UniqueName: \"kubernetes.io/projected/f1912e5d-3dc3-4401-b51d-aa9a52e89c10-kube-api-access-t79p7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:26:55.758024 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.757997 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" Oct 31 21:26:55.758024 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.758003 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h" event={"ID":"f1912e5d-3dc3-4401-b51d-aa9a52e89c10","Type":"ContainerDied","Data":"2768c945b230b62c858b6c4b7a4886c4e0e6140b43fa1550ac34fb8390f5056f"} Oct 31 21:26:55.758506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.758082 4294 scope.go:122] "RemoveContainer" containerID="ed1e2109940f67d3032b390ec39dc2cbcd4cbb91a6a04ad5e1863f13660ebe00" Oct 31 21:26:55.817550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.817516 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h"] Oct 31 21:26:55.829465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.829438 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-353db9f5-84c6956f98-8rl2h"] Oct 31 21:26:55.983274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:26:55.983241 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1912e5d-3dc3-4401-b51d-aa9a52e89c10" path="/var/lib/kubelet/pods/f1912e5d-3dc3-4401-b51d-aa9a52e89c10/volumes" Oct 31 21:27:00.030760 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.030731 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf"] Oct 31 21:27:00.031247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.030928 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" podUID="0fb8afc2-9081-434d-8a1c-2450a8e47cd4" containerName="maas-api" containerID="cri-o://2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556" gracePeriod=30 Oct 31 21:27:00.276266 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.276235 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:27:00.380894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.380813 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-metrics-certs\") pod \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " Oct 31 21:27:00.380894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.380864 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-maas-api-tls\") pod \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " Oct 31 21:27:00.381116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.380923 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-kube-api-access-jd5t5\" (UniqueName: \"kubernetes.io/projected/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-kube-api-access-jd5t5\") pod \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " Oct 31 21:27:00.381116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.380969 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-openshift-service-ca\") pod \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\" (UID: \"0fb8afc2-9081-434d-8a1c-2450a8e47cd4\") " Oct 31 21:27:00.381408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.381382 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-openshift-service-ca" pod "0fb8afc2-9081-434d-8a1c-2450a8e47cd4" (UID: "0fb8afc2-9081-434d-8a1c-2450a8e47cd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:00.382949 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.382919 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-kube-api-access-jd5t5" pod "0fb8afc2-9081-434d-8a1c-2450a8e47cd4" (UID: "0fb8afc2-9081-434d-8a1c-2450a8e47cd4"). InnerVolumeSpecName "kube-api-access-jd5t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:00.383048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.382954 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-metrics-certs" pod "0fb8afc2-9081-434d-8a1c-2450a8e47cd4" (UID: "0fb8afc2-9081-434d-8a1c-2450a8e47cd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:27:00.383048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.382973 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-maas-api-tls" pod "0fb8afc2-9081-434d-8a1c-2450a8e47cd4" (UID: "0fb8afc2-9081-434d-8a1c-2450a8e47cd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:27:00.481998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.481963 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd5t5\" (UniqueName: \"kubernetes.io/projected/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-kube-api-access-jd5t5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.481998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.481995 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.482200 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.482008 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.482200 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.482020 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fb8afc2-9081-434d-8a1c-2450a8e47cd4-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:00.781764 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.781732 4294 generic.go:356] "Generic (PLEG): container finished" podID="0fb8afc2-9081-434d-8a1c-2450a8e47cd4" containerID="2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556" exitCode=0 Oct 31 21:27:00.781939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.781779 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" Oct 31 21:27:00.781939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.781823 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" event={"ID":"0fb8afc2-9081-434d-8a1c-2450a8e47cd4","Type":"ContainerDied","Data":"2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556"} Oct 31 21:27:00.781939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.781881 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf" event={"ID":"0fb8afc2-9081-434d-8a1c-2450a8e47cd4","Type":"ContainerDied","Data":"91665e7f1f72f2fcf34c8f159717fe16f809cef6cd04e5eb3107a556ab49cc2c"} Oct 31 21:27:00.781939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.781909 4294 scope.go:122] "RemoveContainer" containerID="2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556" Oct 31 21:27:00.801209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.801190 4294 scope.go:122] "RemoveContainer" containerID="2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556" Oct 31 21:27:00.801545 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:27:00.801524 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556\": container with ID starting with 2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556 not found: ID does not exist" containerID="2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556" Oct 31 21:27:00.801618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.801558 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556"} err="failed to get container status \"2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556\": rpc error: code = NotFound desc = could not find container \"2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556\": container with ID starting with 2ee4f30c124bc112dab58a533a63898d4ed5b140e7150d3cab87e1e49a881556 not found: ID does not exist" Oct 31 21:27:00.833036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.833010 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf"] Oct 31 21:27:00.845007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:00.844982 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ac0736b6-6848b48848-gskbf"] Oct 31 21:27:01.309390 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:27:01.309355 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:27:01.984126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:01.984092 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb8afc2-9081-434d-8a1c-2450a8e47cd4" path="/var/lib/kubelet/pods/0fb8afc2-9081-434d-8a1c-2450a8e47cd4/volumes" Oct 31 21:27:02.545109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.545074 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw"] Oct 31 21:27:02.545595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.545327 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" podUID="c2834b9c-0d60-442f-8c38-6daace8e9dc5" containerName="maas-api" containerID="cri-o://7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91" gracePeriod=30 Oct 31 21:27:02.792770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.792734 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:27:02.793096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.793076 4294 generic.go:356] "Generic (PLEG): container finished" podID="c2834b9c-0d60-442f-8c38-6daace8e9dc5" containerID="7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91" exitCode=0 Oct 31 21:27:02.793172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.793142 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" event={"ID":"c2834b9c-0d60-442f-8c38-6daace8e9dc5","Type":"ContainerDied","Data":"7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91"} Oct 31 21:27:02.793211 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.793188 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" event={"ID":"c2834b9c-0d60-442f-8c38-6daace8e9dc5","Type":"ContainerDied","Data":"78bc580f0d89c1f2d406b8d9a643ebd9b395a9a01fda3a8cea734d786847fdca"} Oct 31 21:27:02.793243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.793210 4294 scope.go:122] "RemoveContainer" containerID="7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91" Oct 31 21:27:02.805492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.804687 4294 scope.go:122] "RemoveContainer" containerID="7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91" Oct 31 21:27:02.805615 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:27:02.805589 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91\": container with ID starting with 7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91 not found: ID does not exist" containerID="7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91" Oct 31 21:27:02.805703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.805629 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-metrics-certs\") pod \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " Oct 31 21:27:02.805703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.805660 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2834b9c-0d60-442f-8c38-6daace8e9dc5-kube-api-access-k4gpz\" (UniqueName: \"kubernetes.io/projected/c2834b9c-0d60-442f-8c38-6daace8e9dc5-kube-api-access-k4gpz\") pod \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " Oct 31 21:27:02.805703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.805627 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91"} err="failed to get container status \"7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91\": rpc error: code = NotFound desc = could not find container \"7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91\": container with ID starting with 7d7c18878140f264691f910d1e8b13708a59122dfe40e3b50eb3354263322f91 not found: ID does not exist" Oct 31 21:27:02.805899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.805707 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls\") pod \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " Oct 31 21:27:02.805899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.805748 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2834b9c-0d60-442f-8c38-6daace8e9dc5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2834b9c-0d60-442f-8c38-6daace8e9dc5-openshift-service-ca\") pod \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\" (UID: \"c2834b9c-0d60-442f-8c38-6daace8e9dc5\") " Oct 31 21:27:02.806658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.806629 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2834b9c-0d60-442f-8c38-6daace8e9dc5-openshift-service-ca" pod "c2834b9c-0d60-442f-8c38-6daace8e9dc5" (UID: "c2834b9c-0d60-442f-8c38-6daace8e9dc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:27:02.808610 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.808587 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2834b9c-0d60-442f-8c38-6daace8e9dc5-kube-api-access-k4gpz" pod "c2834b9c-0d60-442f-8c38-6daace8e9dc5" (UID: "c2834b9c-0d60-442f-8c38-6daace8e9dc5"). InnerVolumeSpecName "kube-api-access-k4gpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:27:02.809372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.809346 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-metrics-certs" pod "c2834b9c-0d60-442f-8c38-6daace8e9dc5" (UID: "c2834b9c-0d60-442f-8c38-6daace8e9dc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:27:02.809549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.809528 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls" pod "c2834b9c-0d60-442f-8c38-6daace8e9dc5" (UID: "c2834b9c-0d60-442f-8c38-6daace8e9dc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:27:02.907026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.906998 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:02.907026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.907024 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4gpz\" (UniqueName: \"kubernetes.io/projected/c2834b9c-0d60-442f-8c38-6daace8e9dc5-kube-api-access-k4gpz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:02.907223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.907035 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2834b9c-0d60-442f-8c38-6daace8e9dc5-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:02.907223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:02.907044 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2834b9c-0d60-442f-8c38-6daace8e9dc5-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:27:03.239038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.239002 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl"] Oct 31 21:27:03.240040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240014 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f1912e5d-3dc3-4401-b51d-aa9a52e89c10" containerName="maas-api" Oct 31 21:27:03.240158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240042 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1912e5d-3dc3-4401-b51d-aa9a52e89c10" containerName="maas-api" Oct 31 21:27:03.240158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240058 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0fb8afc2-9081-434d-8a1c-2450a8e47cd4" containerName="maas-api" Oct 31 21:27:03.240158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240066 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb8afc2-9081-434d-8a1c-2450a8e47cd4" containerName="maas-api" Oct 31 21:27:03.240158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240083 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c2834b9c-0d60-442f-8c38-6daace8e9dc5" containerName="maas-api" Oct 31 21:27:03.240158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240091 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2834b9c-0d60-442f-8c38-6daace8e9dc5" containerName="maas-api" Oct 31 21:27:03.240346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240192 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1912e5d-3dc3-4401-b51d-aa9a52e89c10" containerName="maas-api" Oct 31 21:27:03.240346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240205 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb8afc2-9081-434d-8a1c-2450a8e47cd4" containerName="maas-api" Oct 31 21:27:03.240346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.240215 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2834b9c-0d60-442f-8c38-6daace8e9dc5" containerName="maas-api" Oct 31 21:27:03.245405 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.244882 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.268729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.268708 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-cc45a7-openshift-default-dockercfg-dbkx4\"" Oct 31 21:27:03.282709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.282680 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl"] Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411162 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411208 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411490 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngf5r\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-kube-api-access-ngf5r\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411545 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411662 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411701 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411776 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411827 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.412020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.411862 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513121 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513162 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513279 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513356 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513381 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngf5r\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-kube-api-access-ngf5r\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513400 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513432 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513462 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513514 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513556 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513731 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513848 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513822 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.513927 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513905 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.514051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.513958 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.515478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.515456 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.515618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.515602 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.539076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.539047 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.539234 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.539151 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngf5r\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-kube-api-access-ngf5r\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.559381 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.559356 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:03.799738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.799653 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw" Oct 31 21:27:03.862391 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.862354 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw"] Oct 31 21:27:03.872124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.872096 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl"] Oct 31 21:27:03.873629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.873606 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-7ef0e96a-56dc895d7d-fs5nw"] Oct 31 21:27:03.874179 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:27:03.874154 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4662f0_9136_44ad_8d6d_fe0ea091fafe.slice/crio-7f418c14df59f0e5c2f1dba9200acf368da1c4a67f2c81e8821b0c9ae7fb235f WatchSource:0}: Error finding container 7f418c14df59f0e5c2f1dba9200acf368da1c4a67f2c81e8821b0c9ae7fb235f: Status 404 returned error can't find the container with id 7f418c14df59f0e5c2f1dba9200acf368da1c4a67f2c81e8821b0c9ae7fb235f Oct 31 21:27:03.876583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.876555 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:27:03.876668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.876611 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:27:03.876668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.876639 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:27:03.989320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:03.989269 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2834b9c-0d60-442f-8c38-6daace8e9dc5" path="/var/lib/kubelet/pods/c2834b9c-0d60-442f-8c38-6daace8e9dc5/volumes" Oct 31 21:27:04.806730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:04.806693 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" event={"ID":"de4662f0-9136-44ad-8d6d-fe0ea091fafe","Type":"ContainerStarted","Data":"65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e"} Oct 31 21:27:04.807139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:04.806749 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" event={"ID":"de4662f0-9136-44ad-8d6d-fe0ea091fafe","Type":"ContainerStarted","Data":"7f418c14df59f0e5c2f1dba9200acf368da1c4a67f2c81e8821b0c9ae7fb235f"} Oct 31 21:27:04.866808 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:04.866759 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" podStartSLOduration=1.866745026 podStartE2EDuration="1.866745026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:04.865172987 +0000 UTC m=+664.990280218" watchObservedRunningTime="2026-10-31 21:27:04.866745026 +0000 UTC m=+664.991852296" Oct 31 21:27:05.190242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.190163 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9"] Oct 31 21:27:05.193797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.193781 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.208535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.208506 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-4b5989-serving-cert\"" Oct 31 21:27:05.208673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.208661 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 21:27:05.208727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.208713 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-kmnpr\"" Oct 31 21:27:05.221957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.221930 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9"] Oct 31 21:27:05.234222 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.234184 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.234368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.234280 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.234454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.234402 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnmv\" (UniqueName: \"kubernetes.io/projected/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-kube-api-access-cvnmv\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.234528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.234452 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.335202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.335165 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.335430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.335234 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.335430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.335287 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.335430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.335349 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnmv\" (UniqueName: \"kubernetes.io/projected/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-kube-api-access-cvnmv\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.335880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.335847 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.337779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.337758 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-maas-api-tls\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.337918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.337899 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-metrics-certs\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.360055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.360032 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnmv\" (UniqueName: \"kubernetes.io/projected/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-kube-api-access-cvnmv\") pod \"maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.505996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.505967 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:05.560480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.560445 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:05.566851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.566827 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:05.812091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.812012 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:05.812947 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.812913 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:05.813511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:05.813488 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9"] Oct 31 21:27:05.815747 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:27:05.815725 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1162f0_fbbf_4bfb_9bfd_e0977c2e40ea.slice/crio-507219e89515d9be59cd397d9258a9ce809ff38bce9136aa36aba7c8cddd24fe WatchSource:0}: Error finding container 507219e89515d9be59cd397d9258a9ce809ff38bce9136aa36aba7c8cddd24fe: Status 404 returned error can't find the container with id 507219e89515d9be59cd397d9258a9ce809ff38bce9136aa36aba7c8cddd24fe Oct 31 21:27:06.816732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:06.816695 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" event={"ID":"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea","Type":"ContainerStarted","Data":"bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95"} Oct 31 21:27:06.817142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:06.816898 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:06.817142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:06.816939 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" event={"ID":"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea","Type":"ContainerStarted","Data":"507219e89515d9be59cd397d9258a9ce809ff38bce9136aa36aba7c8cddd24fe"} Oct 31 21:27:06.860445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:06.860397 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" podStartSLOduration=1.531957427 podStartE2EDuration="1.860382935s" totalImagesPullingTime="328.425508ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:06.859388398 +0000 UTC m=+666.984495629" watchObservedRunningTime="2026-10-31 21:27:06.860382935 +0000 UTC m=+666.985490164" Oct 31 21:27:11.440658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.440629 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b"] Oct 31 21:27:11.444155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.444139 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.471539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.471508 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-c16684-serving-cert\"" Oct 31 21:27:11.473109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.473086 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b"] Oct 31 21:27:11.530738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.530703 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a0a5c3-a26f-45b1-853f-242b1896dcff-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.530931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.530873 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-metrics-certs\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.530931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.530914 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwqns\" (UniqueName: \"kubernetes.io/projected/83a0a5c3-a26f-45b1-853f-242b1896dcff-kube-api-access-kwqns\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.531026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.530967 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.631722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.631689 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-metrics-certs\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.631722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.631728 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwqns\" (UniqueName: \"kubernetes.io/projected/83a0a5c3-a26f-45b1-853f-242b1896dcff-kube-api-access-kwqns\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.631973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.631768 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.631973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.631807 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a0a5c3-a26f-45b1-853f-242b1896dcff-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.632534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.632510 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a0a5c3-a26f-45b1-853f-242b1896dcff-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.634152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.634131 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-metrics-certs\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.634363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.634344 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.655667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.655636 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwqns\" (UniqueName: \"kubernetes.io/projected/83a0a5c3-a26f-45b1-853f-242b1896dcff-kube-api-access-kwqns\") pod \"maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:11.757330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:11.757273 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:12.059446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:12.059419 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b"] Oct 31 21:27:12.059778 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:27:12.059759 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a0a5c3_a26f_45b1_853f_242b1896dcff.slice/crio-34a788fb7753e08aa5516d54fa1075013850c9e4ef7a3421dda958ac1e08a2ea WatchSource:0}: Error finding container 34a788fb7753e08aa5516d54fa1075013850c9e4ef7a3421dda958ac1e08a2ea: Status 404 returned error can't find the container with id 34a788fb7753e08aa5516d54fa1075013850c9e4ef7a3421dda958ac1e08a2ea Oct 31 21:27:12.832281 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:12.832253 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 21:27:12.848033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:12.848002 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" event={"ID":"83a0a5c3-a26f-45b1-853f-242b1896dcff","Type":"ContainerStarted","Data":"82896a00941302883a2b1c3ae91ffd3cb46d25cfdb756364dc5be85a9fa0cc80"} Oct 31 21:27:12.848201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:12.848057 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" event={"ID":"83a0a5c3-a26f-45b1-853f-242b1896dcff","Type":"ContainerStarted","Data":"34a788fb7753e08aa5516d54fa1075013850c9e4ef7a3421dda958ac1e08a2ea"} Oct 31 21:27:12.848201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:12.848088 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:12.918615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:12.918563 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" podStartSLOduration=1.6369322899999998 podStartE2EDuration="1.918547114s" totalImagesPullingTime="281.614824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:12.91771646 +0000 UTC m=+673.042823690" watchObservedRunningTime="2026-10-31 21:27:12.918547114 +0000 UTC m=+673.043654344" Oct 31 21:27:18.859603 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:18.859573 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 21:27:30.100976 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.100933 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf"] Oct 31 21:27:30.104800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.104775 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.131934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.131899 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"kube-root-ca.crt\"" Oct 31 21:27:30.132090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.131940 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"simulator-w3-106555-kserve-self-signed-certs\"" Oct 31 21:27:30.132090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.131961 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"openshift-service-ca.crt\"" Oct 31 21:27:30.132090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.131940 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"default-dockercfg-g8p4t\"" Oct 31 21:27:30.133675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.133651 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf"] Oct 31 21:27:30.156422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.156392 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-home\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.156560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.156426 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82abe779-421c-476a-8f64-bdecf9dccf3f-tls-certs\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.156560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.156499 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-dshm\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.156560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.156551 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-tmp-dir\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.156721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.156659 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-model-cache\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.156721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.156710 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5qc6\" (UniqueName: \"kubernetes.io/projected/82abe779-421c-476a-8f64-bdecf9dccf3f-kube-api-access-k5qc6\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257131 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-model-cache\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257169 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qc6\" (UniqueName: \"kubernetes.io/projected/82abe779-421c-476a-8f64-bdecf9dccf3f-kube-api-access-k5qc6\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257455 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257232 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-home\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257455 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257257 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82abe779-421c-476a-8f64-bdecf9dccf3f-tls-certs\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257609 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-dshm\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257619 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-model-cache\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257830 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257808 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-home\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.257905 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.257828 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-tmp-dir\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.258132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.258113 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-tmp-dir\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.259740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.259693 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-dshm\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.260522 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.260503 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82abe779-421c-476a-8f64-bdecf9dccf3f-tls-certs\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.280481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.280446 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qc6\" (UniqueName: \"kubernetes.io/projected/82abe779-421c-476a-8f64-bdecf9dccf3f-kube-api-access-k5qc6\") pod \"simulator-w3-106555-kserve-5975f447b-9kpgf\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.421384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.421290 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:30.717426 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.717400 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf"] Oct 31 21:27:30.717982 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:27:30.717959 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82abe779_421c_476a_8f64_bdecf9dccf3f.slice/crio-23f1409705091bd3964665388cf6a0be96aca33b2b570a8d79d669389099e49f WatchSource:0}: Error finding container 23f1409705091bd3964665388cf6a0be96aca33b2b570a8d79d669389099e49f: Status 404 returned error can't find the container with id 23f1409705091bd3964665388cf6a0be96aca33b2b570a8d79d669389099e49f Oct 31 21:27:30.946491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.946414 4294 generic.go:356] "Generic (PLEG): container finished" podID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerID="65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e" exitCode=0 Oct 31 21:27:30.946650 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.946503 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" event={"ID":"de4662f0-9136-44ad-8d6d-fe0ea091fafe","Type":"ContainerDied","Data":"65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e"} Oct 31 21:27:30.946949 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.946924 4294 scope.go:122] "RemoveContainer" containerID="65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e" Oct 31 21:27:30.948441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.948414 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:27:30.948551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.948491 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:27:30.948551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.948531 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:27:30.949449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:30.949420 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" event={"ID":"82abe779-421c-476a-8f64-bdecf9dccf3f","Type":"ContainerStarted","Data":"23f1409705091bd3964665388cf6a0be96aca33b2b570a8d79d669389099e49f"} Oct 31 21:27:31.320278 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:27:31.320214 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:27:31.956172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:31.956146 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" event={"ID":"de4662f0-9136-44ad-8d6d-fe0ea091fafe","Type":"ContainerStarted","Data":"f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b"} Oct 31 21:27:32.559910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:32.559877 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:32.560970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:32.560942 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 21:27:32.561089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:32.560989 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 21:27:32.962343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:32.962237 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" event={"ID":"82abe779-421c-476a-8f64-bdecf9dccf3f","Type":"ContainerStarted","Data":"f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511"} Oct 31 21:27:33.011813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:33.011756 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podStartSLOduration=1.792235021 podStartE2EDuration="3.011742794s" totalImagesPullingTime="1.219507773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:33.010128852 +0000 UTC m=+693.135236084" watchObservedRunningTime="2026-10-31 21:27:33.011742794 +0000 UTC m=+693.136850025" Oct 31 21:27:33.422073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:33.422035 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:33.560265 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:33.560225 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 21:27:33.560697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:33.560288 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 21:27:34.560136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:34.560093 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 21:27:34.560355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:34.560149 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 21:27:35.560209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:35.560178 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 21:27:35.560424 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:35.560394 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 21:27:35.812025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:35.811945 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:36.560054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:36.560015 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 21:27:36.560237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:36.560073 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 21:27:37.563761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:37.563729 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:37.564807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:37.564789 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:27:38.044598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.044560 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s"] Oct 31 21:27:38.050639 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.050616 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.076023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.075967 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c16684\"/\"openshift-service-ca.crt\"" Oct 31 21:27:38.076023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.076002 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s"] Oct 31 21:27:38.076260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.076048 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c16684\"/\"kube-root-ca.crt\"" Oct 31 21:27:38.076401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.076384 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c16684\"/\"default-dockercfg-lk94p\"" Oct 31 21:27:38.078347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.078323 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c16684\"/\"simulator-w2-c16684-kserve-self-signed-certs\"" Oct 31 21:27:38.170784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.170740 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-model-cache\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.170952 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.170889 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpnh\" (UniqueName: \"kubernetes.io/projected/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-kube-api-access-wvpnh\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.170952 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.170944 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-home\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.171031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.170981 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tls-certs\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.171115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.171088 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tmp-dir\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.171225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.171206 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-dshm\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.272709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.272674 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-model-cache\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.272902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.272737 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpnh\" (UniqueName: \"kubernetes.io/projected/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-kube-api-access-wvpnh\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.272902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.272760 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-home\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.272902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.272783 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tls-certs\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.272902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.272799 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tmp-dir\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.272902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.272858 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-dshm\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.273186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.273156 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-model-cache\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.273253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.273179 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-home\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.273409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.273388 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tmp-dir\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.275765 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.275731 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-dshm\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.276009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.275988 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tls-certs\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.296682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.296603 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpnh\" (UniqueName: \"kubernetes.io/projected/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-kube-api-access-wvpnh\") pod \"simulator-w2-c16684-kserve-5dc7686c49-vf79s\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.366715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.366666 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:38.660454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:38.660421 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s"] Oct 31 21:27:38.661287 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:27:38.661260 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f2d31b6_3ec4_4fd2_8621_90b296663dc9.slice/crio-1717b4278ac78f23cc50ba916220eed32f590ec9e42e6869b7900d7da2b2bbf3 WatchSource:0}: Error finding container 1717b4278ac78f23cc50ba916220eed32f590ec9e42e6869b7900d7da2b2bbf3: Status 404 returned error can't find the container with id 1717b4278ac78f23cc50ba916220eed32f590ec9e42e6869b7900d7da2b2bbf3 Oct 31 21:27:39.016921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:39.016889 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" event={"ID":"2f2d31b6-3ec4-4fd2-8621-90b296663dc9","Type":"ContainerStarted","Data":"08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178"} Oct 31 21:27:39.017125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:39.016931 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" event={"ID":"2f2d31b6-3ec4-4fd2-8621-90b296663dc9","Type":"ContainerStarted","Data":"1717b4278ac78f23cc50ba916220eed32f590ec9e42e6869b7900d7da2b2bbf3"} Oct 31 21:27:39.058857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:39.058812 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podStartSLOduration=1.058795862 podStartE2EDuration="1.058795862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:39.057941622 +0000 UTC m=+699.183048857" watchObservedRunningTime="2026-10-31 21:27:39.058795862 +0000 UTC m=+699.183903094" Oct 31 21:27:39.367869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:39.367792 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:40.421816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:40.421785 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:40.433967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:40.433942 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:41.036237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:41.036206 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:27:48.367432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.367391 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:48.382903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.382876 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:48.462965 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.462929 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz"] Oct 31 21:27:48.466883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.466863 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.485811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.485786 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-994336\"/\"default-dockercfg-dp8bn\"" Oct 31 21:27:48.486224 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.486193 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-994336\"/\"simulator-w1-994336-kserve-self-signed-certs\"" Oct 31 21:27:48.486339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.486288 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-994336\"/\"openshift-service-ca.crt\"" Oct 31 21:27:48.487525 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.487504 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz"] Oct 31 21:27:48.494477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.494455 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-994336\"/\"kube-root-ca.crt\"" Oct 31 21:27:48.593766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.593737 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-model-cache\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.593950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.593782 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-home\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.593950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.593805 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-749tm\" (UniqueName: \"kubernetes.io/projected/f2ee6869-0354-4185-840d-adee55cc4168-kube-api-access-749tm\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.594055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.593945 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-dshm\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.594055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.593995 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-tmp-dir\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.594055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.594042 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ee6869-0354-4185-840d-adee55cc4168-tls-certs\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.695754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.695655 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-model-cache\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.695754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.695722 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-home\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.695754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.695749 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-749tm\" (UniqueName: \"kubernetes.io/projected/f2ee6869-0354-4185-840d-adee55cc4168-kube-api-access-749tm\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.696012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.695789 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-dshm\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.696012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.695815 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-tmp-dir\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.696012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.695846 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ee6869-0354-4185-840d-adee55cc4168-tls-certs\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.696789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.696763 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-home\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.698892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.697017 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-tmp-dir\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.698892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.697441 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-model-cache\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.699605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.699583 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-dshm\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.703621 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.703583 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ee6869-0354-4185-840d-adee55cc4168-tls-certs\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.720701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.720671 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-749tm\" (UniqueName: \"kubernetes.io/projected/f2ee6869-0354-4185-840d-adee55cc4168-kube-api-access-749tm\") pod \"simulator-w1-994336-kserve-d4cc9dff4-szsbz\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:48.779568 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:48.779538 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:49.077485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:49.077458 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz"] Oct 31 21:27:49.079586 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:27:49.079563 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2ee6869_0354_4185_840d_adee55cc4168.slice/crio-e720a2596985d10b92a789cdf91cd96d220654766fc251bdab9769bf0ac70b23 WatchSource:0}: Error finding container e720a2596985d10b92a789cdf91cd96d220654766fc251bdab9769bf0ac70b23: Status 404 returned error can't find the container with id e720a2596985d10b92a789cdf91cd96d220654766fc251bdab9769bf0ac70b23 Oct 31 21:27:49.084934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:49.084916 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:27:50.076896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:50.076858 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" event={"ID":"f2ee6869-0354-4185-840d-adee55cc4168","Type":"ContainerStarted","Data":"f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6"} Oct 31 21:27:50.077316 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:50.076907 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" event={"ID":"f2ee6869-0354-4185-840d-adee55cc4168","Type":"ContainerStarted","Data":"e720a2596985d10b92a789cdf91cd96d220654766fc251bdab9769bf0ac70b23"} Oct 31 21:27:50.121460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:50.121411 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podStartSLOduration=2.12139788 podStartE2EDuration="2.12139788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:27:50.120066468 +0000 UTC m=+710.245173694" watchObservedRunningTime="2026-10-31 21:27:50.12139788 +0000 UTC m=+710.246505110" Oct 31 21:27:50.780066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:50.780033 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:58.780559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:58.780520 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:58.812898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:58.812868 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:27:59.137846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:27:59.137763 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:28:01.309405 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:28:01.309370 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:28:12.549437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.549405 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls"] Oct 31 21:28:12.553216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.553196 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.571262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.571241 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls"] Oct 31 21:28:12.581189 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.581170 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c16684\"/\"premium-w2-c16684-kserve-self-signed-certs\"" Oct 31 21:28:12.643871 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.643841 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-model-cache\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.644021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.643885 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gkq\" (UniqueName: \"kubernetes.io/projected/937f9e18-5cf6-4f15-b70e-7c960f57a58d-kube-api-access-v4gkq\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.644021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.643911 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-home\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.644021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.643981 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tmp-dir\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.644147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.644099 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tls-certs\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.644203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.644150 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-dshm\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.746118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.745641 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gkq\" (UniqueName: \"kubernetes.io/projected/937f9e18-5cf6-4f15-b70e-7c960f57a58d-kube-api-access-v4gkq\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.746343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.746167 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-home\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.746647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.746623 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-home\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.746761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.746705 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tmp-dir\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.750319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.747015 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tmp-dir\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.750319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.747105 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tls-certs\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.750319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.747616 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-dshm\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.750319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.747990 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-model-cache\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.750319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.748294 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-model-cache\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.750319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.750318 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tls-certs\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.756786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.756761 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-dshm\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.769637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.769608 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gkq\" (UniqueName: \"kubernetes.io/projected/937f9e18-5cf6-4f15-b70e-7c960f57a58d-kube-api-access-v4gkq\") pod \"premium-w2-c16684-kserve-6c48ffc6d8-8hxls\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:12.866043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:12.865975 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:13.171046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:13.171015 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls"] Oct 31 21:28:13.171943 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:28:13.171920 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937f9e18_5cf6_4f15_b70e_7c960f57a58d.slice/crio-7285f937c30863ae75680e9048989711ca84e781025b2ea204e03b4d958998b7 WatchSource:0}: Error finding container 7285f937c30863ae75680e9048989711ca84e781025b2ea204e03b4d958998b7: Status 404 returned error can't find the container with id 7285f937c30863ae75680e9048989711ca84e781025b2ea204e03b4d958998b7 Oct 31 21:28:13.195548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:13.195518 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" event={"ID":"937f9e18-5cf6-4f15-b70e-7c960f57a58d","Type":"ContainerStarted","Data":"7285f937c30863ae75680e9048989711ca84e781025b2ea204e03b4d958998b7"} Oct 31 21:28:14.202025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:14.201991 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" event={"ID":"937f9e18-5cf6-4f15-b70e-7c960f57a58d","Type":"ContainerStarted","Data":"815e4ef4c4909ca26c35256c7818eebe42d5c3972b23622adbffe302638c49f2"} Oct 31 21:28:14.248480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:14.248428 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podStartSLOduration=2.248413968 podStartE2EDuration="2.248413968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:28:14.24728195 +0000 UTC m=+734.372389181" watchObservedRunningTime="2026-10-31 21:28:14.248413968 +0000 UTC m=+734.373521199" Oct 31 21:28:14.867100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:14.867064 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:21.021123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.021089 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk"] Oct 31 21:28:21.024633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.024608 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.063738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.063708 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk"] Oct 31 21:28:21.073587 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.073565 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-994336\"/\"premium-w1-994336-kserve-self-signed-certs\"" Oct 31 21:28:21.137784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.137753 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tls-certs\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.137784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.137799 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-dshm\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.138037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.137861 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5p7j\" (UniqueName: \"kubernetes.io/projected/a41c1a24-0bf9-42cf-aaca-ef77b311c000-kube-api-access-f5p7j\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.138037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.137894 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-home\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.138037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.137913 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-model-cache\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.138037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.138022 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tmp-dir\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239062 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239014 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tmp-dir\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tls-certs\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239135 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-dshm\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239172 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5p7j\" (UniqueName: \"kubernetes.io/projected/a41c1a24-0bf9-42cf-aaca-ef77b311c000-kube-api-access-f5p7j\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239204 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-home\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239226 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-model-cache\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239561 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239514 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tmp-dir\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239596 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-model-cache\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.239864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.239834 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-home\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.242355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.242328 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tls-certs\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.242558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.242536 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-dshm\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.296003 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.295923 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5p7j\" (UniqueName: \"kubernetes.io/projected/a41c1a24-0bf9-42cf-aaca-ef77b311c000-kube-api-access-f5p7j\") pod \"premium-w1-994336-kserve-6bfc5d4c67-pvflk\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.336739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.336706 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:21.647078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:21.647051 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk"] Oct 31 21:28:21.648950 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:28:21.648914 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda41c1a24_0bf9_42cf_aaca_ef77b311c000.slice/crio-abfee9ee58058561cdeae4413e9c42be43d4d6d58910593a1d0914ceadbd2b3d WatchSource:0}: Error finding container abfee9ee58058561cdeae4413e9c42be43d4d6d58910593a1d0914ceadbd2b3d: Status 404 returned error can't find the container with id abfee9ee58058561cdeae4413e9c42be43d4d6d58910593a1d0914ceadbd2b3d Oct 31 21:28:22.260323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:22.256539 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" event={"ID":"a41c1a24-0bf9-42cf-aaca-ef77b311c000","Type":"ContainerStarted","Data":"33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa"} Oct 31 21:28:22.260323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:22.256604 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" event={"ID":"a41c1a24-0bf9-42cf-aaca-ef77b311c000","Type":"ContainerStarted","Data":"abfee9ee58058561cdeae4413e9c42be43d4d6d58910593a1d0914ceadbd2b3d"} Oct 31 21:28:22.308622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:22.308557 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podStartSLOduration=2.308526299 podStartE2EDuration="2.308526299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:28:22.307745374 +0000 UTC m=+742.432852603" watchObservedRunningTime="2026-10-31 21:28:22.308526299 +0000 UTC m=+742.433633538" Oct 31 21:28:22.342373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:22.339380 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:22.866432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:22.866402 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:22.885194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:22.885172 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:23.272953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:23.272923 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:28:31.310965 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:28:31.310920 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:28:31.337269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:31.337240 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:31.350166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:31.350138 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:28:32.317239 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:28:32.317205 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:29:01.309617 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:29:01.309541 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:29:27.422688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.422652 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr"] Oct 31 21:29:27.426228 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.426213 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.445534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.445512 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-106555-openshift-default-dockercfg-v5bxg\"" Oct 31 21:29:27.458066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.458042 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr"] Oct 31 21:29:27.483875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.483851 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.483975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.483902 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.484016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.483977 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.484016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.484003 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.484086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.484022 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969dz\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-kube-api-access-969dz\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.484086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.484047 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.484180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.484161 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.484215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.484188 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.484246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.484216 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585451 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585479 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585496 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969dz\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-kube-api-access-969dz\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585516 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585559 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585583 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585616 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585693 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585755 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.585999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.585983 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.586031 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.586116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.586096 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.586169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.586130 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.586352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.586296 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.588035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.588016 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.588165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.588146 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.609034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.609009 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.609246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.609228 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-969dz\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-kube-api-access-969dz\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:27.738576 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:27.738555 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:28.053111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.053082 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr"] Oct 31 21:29:28.054031 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:29:28.054002 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b9fc8b4_6465_412a_84d8_3cef75fb8ffd.slice/crio-096e67911d2e7ecd6efbb7c27fb53e169c3458edc88c30e99f10589bd0342b99 WatchSource:0}: Error finding container 096e67911d2e7ecd6efbb7c27fb53e169c3458edc88c30e99f10589bd0342b99: Status 404 returned error can't find the container with id 096e67911d2e7ecd6efbb7c27fb53e169c3458edc88c30e99f10589bd0342b99 Oct 31 21:29:28.057179 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.057133 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:29:28.057454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.057426 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:29:28.057536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.057496 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:29:28.602765 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.602728 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" event={"ID":"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd","Type":"ContainerStarted","Data":"c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143"} Oct 31 21:29:28.603131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.602796 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" event={"ID":"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd","Type":"ContainerStarted","Data":"096e67911d2e7ecd6efbb7c27fb53e169c3458edc88c30e99f10589bd0342b99"} Oct 31 21:29:28.673056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.673010 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podStartSLOduration=1.672997542 podStartE2EDuration="1.672997542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:28.659912696 +0000 UTC m=+808.785019926" watchObservedRunningTime="2026-10-31 21:29:28.672997542 +0000 UTC m=+808.798104780" Oct 31 21:29:28.739235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:28.739204 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:29.741113 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:29.741083 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 21:29:29.741519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:29.741130 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 21:29:30.739900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:30.739837 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 21:29:30.740084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:30.739916 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 21:29:31.381327 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:29:31.372590 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:29:31.744324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:31.740653 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 21:29:31.744324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:31.740717 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 21:29:32.745329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:32.743483 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 21:29:32.745329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:32.743544 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 21:29:33.738867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:33.738830 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 21:29:33.739033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:33.738885 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 21:29:33.923259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:33.923222 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8"] Oct 31 21:29:33.937140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:33.937107 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:33.951961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:33.951930 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8"] Oct 31 21:29:33.960274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:33.960243 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"e2e-conflict-a-08e7b19c-kserve-self-signed-certs\"" Oct 31 21:29:34.029047 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.028961 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tmp-dir\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.029205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.029097 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-dshm\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.029205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.029147 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-model-cache\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.029205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.029171 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tls-certs\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.029395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.029326 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-home\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.029395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.029365 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msrl4\" (UniqueName: \"kubernetes.io/projected/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-kube-api-access-msrl4\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.130703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.130658 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msrl4\" (UniqueName: \"kubernetes.io/projected/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-kube-api-access-msrl4\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.130890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.130725 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tmp-dir\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.130890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.130815 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-dshm\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.130890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.130870 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-model-cache\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.131105 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.130903 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tls-certs\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.131105 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.130978 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-home\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.131482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.131448 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-home\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.131913 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.131891 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-model-cache\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.132277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.132255 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tmp-dir\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.136322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.134989 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tls-certs\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.136322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.135041 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-dshm\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.181324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.180905 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msrl4\" (UniqueName: \"kubernetes.io/projected/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-kube-api-access-msrl4\") pod \"e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.252811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.252775 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:34.586771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.586744 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8"] Oct 31 21:29:34.587928 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:29:34.587895 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd383ab3b_e37e_4bdf_9e81_36b8ee16070e.slice/crio-15aead26984c4efb8edf243d7c9dea4b6313d7eb46b12e6c6c5b0bfa5b236a23 WatchSource:0}: Error finding container 15aead26984c4efb8edf243d7c9dea4b6313d7eb46b12e6c6c5b0bfa5b236a23: Status 404 returned error can't find the container with id 15aead26984c4efb8edf243d7c9dea4b6313d7eb46b12e6c6c5b0bfa5b236a23 Oct 31 21:29:34.648745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.648715 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" event={"ID":"d383ab3b-e37e-4bdf-9e81-36b8ee16070e","Type":"ContainerStarted","Data":"15aead26984c4efb8edf243d7c9dea4b6313d7eb46b12e6c6c5b0bfa5b236a23"} Oct 31 21:29:34.745323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.739961 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 21:29:34.745323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:34.740020 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 21:29:35.655151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:35.655112 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" event={"ID":"d383ab3b-e37e-4bdf-9e81-36b8ee16070e","Type":"ContainerStarted","Data":"4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854"} Oct 31 21:29:35.700137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:35.700076 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" podStartSLOduration=2.700056981 podStartE2EDuration="2.700056981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:35.699434231 +0000 UTC m=+815.824541462" watchObservedRunningTime="2026-10-31 21:29:35.700056981 +0000 UTC m=+815.825164214" Oct 31 21:29:35.743073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:35.743043 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:35.743347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:35.743329 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:35.743915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:35.743896 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:29:36.253162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:36.253133 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:44.254569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:44.254530 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:44.268381 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:44.268350 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:44.745340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:44.745294 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:29:50.303505 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.303467 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs"] Oct 31 21:29:50.307438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.307413 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.325167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.325139 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs"] Oct 31 21:29:50.335283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.335242 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"default-dockercfg-tdnws\"" Oct 31 21:29:50.335462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.335286 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"distinct-w0-4b5989-kserve-self-signed-certs\"" Oct 31 21:29:50.416832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.416796 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"kube-root-ca.crt\"" Oct 31 21:29:50.416832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.416797 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4b5989\"/\"openshift-service-ca.crt\"" Oct 31 21:29:50.428851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.428814 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tmp-dir\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.429015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.428859 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-home\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.429015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.428939 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hbk5\" (UniqueName: \"kubernetes.io/projected/51a9eca7-b8a4-4428-a645-359a8bddb7b2-kube-api-access-6hbk5\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.429135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.429050 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-model-cache\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.429193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.429170 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tls-certs\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.429243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.429214 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-dshm\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.530599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.530557 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tls-certs\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.530797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.530606 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-dshm\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.530797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.530689 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tmp-dir\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.530797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.530718 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-home\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.530797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.530748 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hbk5\" (UniqueName: \"kubernetes.io/projected/51a9eca7-b8a4-4428-a645-359a8bddb7b2-kube-api-access-6hbk5\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.531012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.530823 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-model-cache\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.531183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.531160 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-home\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.531247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.531197 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tmp-dir\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.531471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.531448 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-model-cache\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.533015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.532989 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-dshm\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.533179 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.533162 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tls-certs\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.554677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.554572 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hbk5\" (UniqueName: \"kubernetes.io/projected/51a9eca7-b8a4-4428-a645-359a8bddb7b2-kube-api-access-6hbk5\") pod \"distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.620825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.620785 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:50.932667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:50.932634 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs"] Oct 31 21:29:50.934270 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:29:50.934243 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51a9eca7_b8a4_4428_a645_359a8bddb7b2.slice/crio-f5bfb591a9fb5690020fb926778a5b8da7a4f00ae0fc2fbbd381e85957dab5b7 WatchSource:0}: Error finding container f5bfb591a9fb5690020fb926778a5b8da7a4f00ae0fc2fbbd381e85957dab5b7: Status 404 returned error can't find the container with id f5bfb591a9fb5690020fb926778a5b8da7a4f00ae0fc2fbbd381e85957dab5b7 Oct 31 21:29:51.772297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.772264 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" event={"ID":"51a9eca7-b8a4-4428-a645-359a8bddb7b2","Type":"ContainerStarted","Data":"62dfca15931dda99286d518bd4308ff462a3cfee4e522de5eeee7ac79bfcfef7"} Oct 31 21:29:51.772695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.772323 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" event={"ID":"51a9eca7-b8a4-4428-a645-359a8bddb7b2","Type":"ContainerStarted","Data":"f5bfb591a9fb5690020fb926778a5b8da7a4f00ae0fc2fbbd381e85957dab5b7"} Oct 31 21:29:51.783159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.783129 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6"] Oct 31 21:29:51.786918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.786897 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.803972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.803947 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c16684\"/\"distinct-w2-c16684-kserve-self-signed-certs\"" Oct 31 21:29:51.806152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.806121 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6"] Oct 31 21:29:51.820222 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.820175 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podStartSLOduration=1.8201601809999999 podStartE2EDuration="1.820160181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:51.818599848 +0000 UTC m=+831.943707079" watchObservedRunningTime="2026-10-31 21:29:51.820160181 +0000 UTC m=+831.945267410" Oct 31 21:29:51.846248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.846216 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-home\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.846391 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.846273 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322330f-18fb-405a-9af5-2fa33e9094c0-tls-certs\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.846446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.846405 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-dshm\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.846446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.846437 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-model-cache\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.846524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.846465 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58kzp\" (UniqueName: \"kubernetes.io/projected/e322330f-18fb-405a-9af5-2fa33e9094c0-kube-api-access-58kzp\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.846566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.846542 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-tmp-dir\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.947732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.947690 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-home\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.947732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.947736 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322330f-18fb-405a-9af5-2fa33e9094c0-tls-certs\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.947996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.947766 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-dshm\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.947996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.947780 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-model-cache\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.947996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.947804 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58kzp\" (UniqueName: \"kubernetes.io/projected/e322330f-18fb-405a-9af5-2fa33e9094c0-kube-api-access-58kzp\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.947996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.947844 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-tmp-dir\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.948291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.948161 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-home\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.948291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.948261 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-tmp-dir\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.948291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.948271 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-model-cache\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.950509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.950481 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-dshm\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.951715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.951687 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322330f-18fb-405a-9af5-2fa33e9094c0-tls-certs\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:51.976082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:51.976014 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58kzp\" (UniqueName: \"kubernetes.io/projected/e322330f-18fb-405a-9af5-2fa33e9094c0-kube-api-access-58kzp\") pod \"distinct-w2-c16684-kserve-544c778c55-lcrw6\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:52.099433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:52.099351 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:29:52.398863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:52.398832 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6"] Oct 31 21:29:52.400488 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:29:52.400463 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode322330f_18fb_405a_9af5_2fa33e9094c0.slice/crio-79524ebc858710a03371fcbc6038f17bd617ec6599bffc61eb9cc6675c62e110 WatchSource:0}: Error finding container 79524ebc858710a03371fcbc6038f17bd617ec6599bffc61eb9cc6675c62e110: Status 404 returned error can't find the container with id 79524ebc858710a03371fcbc6038f17bd617ec6599bffc61eb9cc6675c62e110 Oct 31 21:29:52.621843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:52.621808 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:29:52.779159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:52.779119 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" event={"ID":"e322330f-18fb-405a-9af5-2fa33e9094c0","Type":"ContainerStarted","Data":"280d59643a65b156403084746f9a87368dc772271dbaa51ef5c3f078c7bd279a"} Oct 31 21:29:52.779591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:52.779166 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" event={"ID":"e322330f-18fb-405a-9af5-2fa33e9094c0","Type":"ContainerStarted","Data":"79524ebc858710a03371fcbc6038f17bd617ec6599bffc61eb9cc6675c62e110"} Oct 31 21:29:52.843117 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:52.843063 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podStartSLOduration=1.843048603 podStartE2EDuration="1.843048603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:29:52.841549829 +0000 UTC m=+832.966657059" watchObservedRunningTime="2026-10-31 21:29:52.843048603 +0000 UTC m=+832.968155833" Oct 31 21:29:53.102137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:29:53.099527 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:30:00.165235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.165200 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z"] Oct 31 21:30:00.170255 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.170207 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.202125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.201771 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:30:00.206168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.206143 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp"] Oct 31 21:30:00.232688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.232655 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z"] Oct 31 21:30:00.238422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.238389 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp"] Oct 31 21:30:00.255435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.248440 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.256086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.249045 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lqk\" (UniqueName: \"kubernetes.io/projected/6bab82df-2cbd-401a-8095-a38ca60b9ebc-kube-api-access-q7lqk\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.256704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.256680 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab82df-2cbd-401a-8095-a38ca60b9ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.358007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.357973 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwqsx\" (UniqueName: \"kubernetes.io/projected/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-kube-api-access-gwqsx\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.358177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.358039 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.358177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.358108 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab82df-2cbd-401a-8095-a38ca60b9ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.358296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.358198 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lqk\" (UniqueName: \"kubernetes.io/projected/6bab82df-2cbd-401a-8095-a38ca60b9ebc-kube-api-access-q7lqk\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.365335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.359287 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab82df-2cbd-401a-8095-a38ca60b9ebc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.459367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.455679 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lqk\" (UniqueName: \"kubernetes.io/projected/6bab82df-2cbd-401a-8095-a38ca60b9ebc-kube-api-access-q7lqk\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.459582 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.459012 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwqsx\" (UniqueName: \"kubernetes.io/projected/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-kube-api-access-gwqsx\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.459707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.459631 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.460430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.460406 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.524128 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.524090 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwqsx\" (UniqueName: \"kubernetes.io/projected/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-kube-api-access-gwqsx\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.567625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.567592 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:00.625602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.625509 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:30:00.658014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.657983 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:30:00.666099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.666071 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:00.881333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:00.881032 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:30:01.013415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:01.013368 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z"] Oct 31 21:30:01.020649 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:30:01.020596 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bab82df_2cbd_401a_8095_a38ca60b9ebc.slice/crio-1009319cedcf7c17974c04405a68e406adbd573cb04510519b4da012f5d326bb WatchSource:0}: Error finding container 1009319cedcf7c17974c04405a68e406adbd573cb04510519b4da012f5d326bb: Status 404 returned error can't find the container with id 1009319cedcf7c17974c04405a68e406adbd573cb04510519b4da012f5d326bb Oct 31 21:30:01.142707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:01.142670 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp"] Oct 31 21:30:01.143378 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:30:01.143354 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed8c7c1_b867_42a6_8cd7_d2c0362c0336.slice/crio-aace80ae6eba5507d3d987ddd78f3127f4018eb0d0c0678e0a08fabf925f8ccd WatchSource:0}: Error finding container aace80ae6eba5507d3d987ddd78f3127f4018eb0d0c0678e0a08fabf925f8ccd: Status 404 returned error can't find the container with id aace80ae6eba5507d3d987ddd78f3127f4018eb0d0c0678e0a08fabf925f8ccd Oct 31 21:30:01.309611 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:01.309535 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:30:01.867643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:01.867607 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" event={"ID":"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336","Type":"ContainerStarted","Data":"aace80ae6eba5507d3d987ddd78f3127f4018eb0d0c0678e0a08fabf925f8ccd"} Oct 31 21:30:01.868953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:01.868922 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" event={"ID":"6bab82df-2cbd-401a-8095-a38ca60b9ebc","Type":"ContainerStarted","Data":"1009319cedcf7c17974c04405a68e406adbd573cb04510519b4da012f5d326bb"} Oct 31 21:30:02.100410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:02.100380 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:30:02.112480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:02.112461 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:30:02.900114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:02.899897 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:30:04.895694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:04.895627 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" event={"ID":"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336","Type":"ContainerStarted","Data":"14f288b3cf9fd62bfd57fb85ed2a44892d8b0be8b7909caa54b060ca13142d18"} Oct 31 21:30:04.897121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:04.897086 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" event={"ID":"6bab82df-2cbd-401a-8095-a38ca60b9ebc","Type":"ContainerStarted","Data":"bfa8aa95564b1ea56c5d70162080d15439b01777c366bc5eb0cf82b07f7c6d24"} Oct 31 21:30:04.938774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:04.938735 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" podStartSLOduration=1.756126932 podStartE2EDuration="4.938722116s" totalImagesPullingTime="3.182595184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:04.937503768 +0000 UTC m=+845.062611001" watchObservedRunningTime="2026-10-31 21:30:04.938722116 +0000 UTC m=+845.063829347" Oct 31 21:30:04.980546 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:04.980498 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" podStartSLOduration=1.672665717 podStartE2EDuration="4.980483019s" totalImagesPullingTime="3.307817302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:04.978170225 +0000 UTC m=+845.103277459" watchObservedRunningTime="2026-10-31 21:30:04.980483019 +0000 UTC m=+845.105590252" Oct 31 21:30:05.903383 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:05.903347 4294 generic.go:356] "Generic (PLEG): container finished" podID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerID="14f288b3cf9fd62bfd57fb85ed2a44892d8b0be8b7909caa54b060ca13142d18" exitCode=60 Oct 31 21:30:05.903817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:05.903432 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" event={"ID":"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336","Type":"ContainerDied","Data":"14f288b3cf9fd62bfd57fb85ed2a44892d8b0be8b7909caa54b060ca13142d18"} Oct 31 21:30:05.903817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:05.903659 4294 scope.go:122] "RemoveContainer" containerID="14f288b3cf9fd62bfd57fb85ed2a44892d8b0be8b7909caa54b060ca13142d18" Oct 31 21:30:05.905060 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:05.905040 4294 generic.go:356] "Generic (PLEG): container finished" podID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerID="bfa8aa95564b1ea56c5d70162080d15439b01777c366bc5eb0cf82b07f7c6d24" exitCode=60 Oct 31 21:30:05.905173 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:05.905123 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" event={"ID":"6bab82df-2cbd-401a-8095-a38ca60b9ebc","Type":"ContainerDied","Data":"bfa8aa95564b1ea56c5d70162080d15439b01777c366bc5eb0cf82b07f7c6d24"} Oct 31 21:30:05.905346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:05.905326 4294 scope.go:122] "RemoveContainer" containerID="bfa8aa95564b1ea56c5d70162080d15439b01777c366bc5eb0cf82b07f7c6d24" Oct 31 21:30:06.912764 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.912681 4294 generic.go:356] "Generic (PLEG): container finished" podID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerID="63d974d07152c3d1fbec8cdec5190e871f9b8726708a786dc63f3332b5243fbd" exitCode=60 Oct 31 21:30:06.913190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.912758 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" event={"ID":"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336","Type":"ContainerDied","Data":"63d974d07152c3d1fbec8cdec5190e871f9b8726708a786dc63f3332b5243fbd"} Oct 31 21:30:06.913190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.912829 4294 scope.go:122] "RemoveContainer" containerID="14f288b3cf9fd62bfd57fb85ed2a44892d8b0be8b7909caa54b060ca13142d18" Oct 31 21:30:06.913190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.913041 4294 scope.go:122] "RemoveContainer" containerID="63d974d07152c3d1fbec8cdec5190e871f9b8726708a786dc63f3332b5243fbd" Oct 31 21:30:06.913365 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:06.913334 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp_odh-ai-gateway-infra(4ed8c7c1-b867-42a6-8cd7-d2c0362c0336)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" Oct 31 21:30:06.915394 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.915369 4294 generic.go:356] "Generic (PLEG): container finished" podID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerID="b912d1b631a5d9bd8cfb66236386d0262e1fe84d1605633c886318966db8d217" exitCode=60 Oct 31 21:30:06.915495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.915430 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" event={"ID":"6bab82df-2cbd-401a-8095-a38ca60b9ebc","Type":"ContainerDied","Data":"b912d1b631a5d9bd8cfb66236386d0262e1fe84d1605633c886318966db8d217"} Oct 31 21:30:06.915762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.915744 4294 scope.go:122] "RemoveContainer" containerID="b912d1b631a5d9bd8cfb66236386d0262e1fe84d1605633c886318966db8d217" Oct 31 21:30:06.916039 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:06.916017 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z_odh-ai-gateway-infra(6bab82df-2cbd-401a-8095-a38ca60b9ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" Oct 31 21:30:06.930131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:06.930113 4294 scope.go:122] "RemoveContainer" containerID="bfa8aa95564b1ea56c5d70162080d15439b01777c366bc5eb0cf82b07f7c6d24" Oct 31 21:30:07.921618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:07.921591 4294 scope.go:122] "RemoveContainer" containerID="63d974d07152c3d1fbec8cdec5190e871f9b8726708a786dc63f3332b5243fbd" Oct 31 21:30:07.921994 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:07.921837 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp_odh-ai-gateway-infra(4ed8c7c1-b867-42a6-8cd7-d2c0362c0336)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" Oct 31 21:30:07.923445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:07.923426 4294 scope.go:122] "RemoveContainer" containerID="b912d1b631a5d9bd8cfb66236386d0262e1fe84d1605633c886318966db8d217" Oct 31 21:30:07.923656 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:07.923639 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z_odh-ai-gateway-infra(6bab82df-2cbd-401a-8095-a38ca60b9ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" Oct 31 21:30:13.722064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.721991 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6"] Oct 31 21:30:13.732887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.732855 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.751781 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.751752 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6"] Oct 31 21:30:13.760702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.760676 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4b5989-openshift-default-dockercfg-gpzdz\"" Oct 31 21:30:13.840925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.840890 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297b98b-9e10-4e72-a322-3e33984deb9e-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.840930 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.840953 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.841052 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvwz\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-kube-api-access-ngvwz\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.841096 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.841117 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.841166 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.841206 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.841347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.841252 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.942801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942764 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297b98b-9e10-4e72-a322-3e33984deb9e-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.942801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942802 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942819 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942843 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvwz\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-kube-api-access-ngvwz\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942887 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942915 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942945 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.942980 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.943037 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.943221 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.943297 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.943415 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.943628 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.943705 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.943636 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297b98b-9e10-4e72-a322-3e33984deb9e-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.945207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.945185 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.945427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.945409 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.966848 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.966817 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:13.967330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:13.967279 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvwz\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-kube-api-access-ngvwz\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:14.046959 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:14.046924 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:14.367731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:14.367705 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6"] Oct 31 21:30:14.368584 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:30:14.368556 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4297b98b_9e10_4e72_a322_3e33984deb9e.slice/crio-fce36a5ef2395d1a712c2ddea65f75e220bb389371555020742a63b3e34027d8 WatchSource:0}: Error finding container fce36a5ef2395d1a712c2ddea65f75e220bb389371555020742a63b3e34027d8: Status 404 returned error can't find the container with id fce36a5ef2395d1a712c2ddea65f75e220bb389371555020742a63b3e34027d8 Oct 31 21:30:14.371284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:14.371248 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:14.371368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:14.371346 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:14.371406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:14.371379 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:14.964607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:14.964568 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" event={"ID":"4297b98b-9e10-4e72-a322-3e33984deb9e","Type":"ContainerStarted","Data":"6ecc70ced6671b2222b244b451be05401c6012afef6612170480d8324507757b"} Oct 31 21:30:14.965066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:14.964620 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" event={"ID":"4297b98b-9e10-4e72-a322-3e33984deb9e","Type":"ContainerStarted","Data":"fce36a5ef2395d1a712c2ddea65f75e220bb389371555020742a63b3e34027d8"} Oct 31 21:30:15.025316 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:15.025254 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podStartSLOduration=2.025240413 podStartE2EDuration="2.025240413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:15.024068846 +0000 UTC m=+855.149176088" watchObservedRunningTime="2026-10-31 21:30:15.025240413 +0000 UTC m=+855.150347644" Oct 31 21:30:15.047418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:15.047390 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:15.049372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:15.049346 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 21:30:15.049507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:15.049395 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 21:30:16.047942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:16.047912 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 21:30:16.048362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:16.047959 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 21:30:17.048034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:17.048000 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 21:30:17.048438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:17.048051 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 21:30:17.999987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:17.996634 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc"] Oct 31 21:30:18.004652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.004600 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.011143 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.011105 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc"] Oct 31 21:30:18.023294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.021504 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c16684-openshift-default-dockercfg-t7nbj\"" Oct 31 21:30:18.047587 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.047558 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 21:30:18.047751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.047650 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109120 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109169 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109184 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109209 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089877af-57d8-4b4a-9c4d-838f74d53ae9-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109234 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109255 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109272 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b9nn\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-kube-api-access-8b9nn\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109317 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.109453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.109331 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210088 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210153 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210175 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210210 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089877af-57d8-4b4a-9c4d-838f74d53ae9-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210251 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210280 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210322 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b9nn\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-kube-api-access-8b9nn\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210374 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.210640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.210399 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.211365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.211205 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.211618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.211575 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089877af-57d8-4b4a-9c4d-838f74d53ae9-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.211618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.211581 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.211745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.211586 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.211745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.211655 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.213098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.213077 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.213195 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.213172 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.234641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.234622 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b9nn\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-kube-api-access-8b9nn\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.235109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.235089 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.328937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.328847 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:18.682608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.682515 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc"] Oct 31 21:30:18.701877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.700448 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:18.701877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.700538 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:18.701877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:18.700603 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:19.005396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.004800 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" event={"ID":"089877af-57d8-4b4a-9c4d-838f74d53ae9","Type":"ContainerStarted","Data":"e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a"} Oct 31 21:30:19.005396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.004862 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" event={"ID":"089877af-57d8-4b4a-9c4d-838f74d53ae9","Type":"ContainerStarted","Data":"bd27042a164be52c742d9c8ad831941a770ca796ed3aa03501649cea6160c0ba"} Oct 31 21:30:19.047267 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.047235 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 21:30:19.047470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.047294 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 21:30:19.050567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.050516 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podStartSLOduration=2.050495904 podStartE2EDuration="2.050495904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:19.049712816 +0000 UTC m=+859.174820066" watchObservedRunningTime="2026-10-31 21:30:19.050495904 +0000 UTC m=+859.175603136" Oct 31 21:30:19.330039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.329956 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:19.331647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.331625 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 21:30:19.331715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:19.331683 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 21:30:20.027322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:20.027278 4294 scope.go:122] "RemoveContainer" containerID="b912d1b631a5d9bd8cfb66236386d0262e1fe84d1605633c886318966db8d217" Oct 31 21:30:20.047629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:20.047606 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 21:30:20.047780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:20.047762 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 21:30:20.329227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:20.329199 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 21:30:20.329398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:20.329252 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 21:30:21.014014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.013447 4294 scope.go:122] "RemoveContainer" containerID="63d974d07152c3d1fbec8cdec5190e871f9b8726708a786dc63f3332b5243fbd" Oct 31 21:30:21.061370 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.060802 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" event={"ID":"6bab82df-2cbd-401a-8095-a38ca60b9ebc","Type":"ContainerStarted","Data":"06f50476adb868ad2d092110e5bbcbe15c06eb105bd58ed2c5d13d624d29f830"} Oct 31 21:30:21.061370 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.061146 4294 scope.go:122] "RemoveContainer" containerID="06f50476adb868ad2d092110e5bbcbe15c06eb105bd58ed2c5d13d624d29f830" Oct 31 21:30:21.062269 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:21.061754 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z_odh-ai-gateway-infra(6bab82df-2cbd-401a-8095-a38ca60b9ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" Oct 31 21:30:21.330296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.329860 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 21:30:21.330296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.329918 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 21:30:21.560146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.558988 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:21.560146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.560005 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:21.563422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:21.563373 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:30:22.080091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.080052 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" event={"ID":"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336","Type":"ContainerStarted","Data":"db22a68a16414bec7f6c39995e6b8470c46c0befe0d3491411b9fdedd927d748"} Oct 31 21:30:22.097454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.097419 4294 generic.go:356] "Generic (PLEG): container finished" podID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerID="06f50476adb868ad2d092110e5bbcbe15c06eb105bd58ed2c5d13d624d29f830" exitCode=60 Oct 31 21:30:22.101010 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.097501 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" event={"ID":"6bab82df-2cbd-401a-8095-a38ca60b9ebc","Type":"ContainerDied","Data":"06f50476adb868ad2d092110e5bbcbe15c06eb105bd58ed2c5d13d624d29f830"} Oct 31 21:30:22.101010 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.097762 4294 scope.go:122] "RemoveContainer" containerID="b912d1b631a5d9bd8cfb66236386d0262e1fe84d1605633c886318966db8d217" Oct 31 21:30:22.101010 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.098271 4294 scope.go:122] "RemoveContainer" containerID="06f50476adb868ad2d092110e5bbcbe15c06eb105bd58ed2c5d13d624d29f830" Oct 31 21:30:22.101010 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:22.098612 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z_odh-ai-gateway-infra(6bab82df-2cbd-401a-8095-a38ca60b9ebc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" Oct 31 21:30:22.141331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.130484 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z"] Oct 31 21:30:22.329893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.329804 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 21:30:22.329893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:22.329866 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 21:30:23.125891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:23.125855 4294 generic.go:356] "Generic (PLEG): container finished" podID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerID="db22a68a16414bec7f6c39995e6b8470c46c0befe0d3491411b9fdedd927d748" exitCode=60 Oct 31 21:30:23.139338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:23.139289 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" event={"ID":"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336","Type":"ContainerDied","Data":"db22a68a16414bec7f6c39995e6b8470c46c0befe0d3491411b9fdedd927d748"} Oct 31 21:30:23.144410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:23.144376 4294 scope.go:122] "RemoveContainer" containerID="63d974d07152c3d1fbec8cdec5190e871f9b8726708a786dc63f3332b5243fbd" Oct 31 21:30:23.145460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:23.145443 4294 scope.go:122] "RemoveContainer" containerID="db22a68a16414bec7f6c39995e6b8470c46c0befe0d3491411b9fdedd927d748" Oct 31 21:30:23.161920 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:23.161644 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp_odh-ai-gateway-infra(4ed8c7c1-b867-42a6-8cd7-d2c0362c0336)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" Oct 31 21:30:23.245412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:23.230260 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp"] Oct 31 21:30:23.337328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:23.334172 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 21:30:23.337328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:23.334228 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 21:30:24.044371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.037463 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:24.122993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.122436 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:24.275571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.275524 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:24.316376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.300069 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:24.316376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.300721 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bab82df-2cbd-401a-8095-a38ca60b9ebc-kube-api-access-q7lqk\" (UniqueName: \"kubernetes.io/projected/6bab82df-2cbd-401a-8095-a38ca60b9ebc-kube-api-access-q7lqk\") pod \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " Oct 31 21:30:24.316376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.300867 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bab82df-2cbd-401a-8095-a38ca60b9ebc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab82df-2cbd-401a-8095-a38ca60b9ebc-maas-api-service-ca\") pod \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\" (UID: \"6bab82df-2cbd-401a-8095-a38ca60b9ebc\") " Oct 31 21:30:24.316376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.315406 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:30:24.317563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.317046 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bab82df-2cbd-401a-8095-a38ca60b9ebc-maas-api-service-ca" pod "6bab82df-2cbd-401a-8095-a38ca60b9ebc" (UID: "6bab82df-2cbd-401a-8095-a38ca60b9ebc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:24.338350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.330606 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 21:30:24.338350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.330655 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 21:30:24.353073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.353025 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bab82df-2cbd-401a-8095-a38ca60b9ebc-kube-api-access-q7lqk" pod "6bab82df-2cbd-401a-8095-a38ca60b9ebc" (UID: "6bab82df-2cbd-401a-8095-a38ca60b9ebc"). InnerVolumeSpecName "kube-api-access-q7lqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:24.419239 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.419172 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bab82df-2cbd-401a-8095-a38ca60b9ebc-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:30:24.419239 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.419212 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7lqk\" (UniqueName: \"kubernetes.io/projected/6bab82df-2cbd-401a-8095-a38ca60b9ebc-kube-api-access-q7lqk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:30:24.788326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.785054 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:24.842209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.842180 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-maas-api-service-ca\") pod \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " Oct 31 21:30:24.842377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.842268 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-kube-api-access-gwqsx\" (UniqueName: \"kubernetes.io/projected/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-kube-api-access-gwqsx\") pod \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\" (UID: \"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336\") " Oct 31 21:30:24.842644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.842617 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-maas-api-service-ca" pod "4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" (UID: "4ed8c7c1-b867-42a6-8cd7-d2c0362c0336"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:30:24.844340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.844316 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-kube-api-access-gwqsx" pod "4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" (UID: "4ed8c7c1-b867-42a6-8cd7-d2c0362c0336"). InnerVolumeSpecName "kube-api-access-gwqsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:30:24.944112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.944032 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:30:24.944112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:24.944058 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwqsx\" (UniqueName: \"kubernetes.io/projected/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336-kube-api-access-gwqsx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:30:25.287799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.287768 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" Oct 31 21:30:25.287799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.287781 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp" event={"ID":"4ed8c7c1-b867-42a6-8cd7-d2c0362c0336","Type":"ContainerDied","Data":"aace80ae6eba5507d3d987ddd78f3127f4018eb0d0c0678e0a08fabf925f8ccd"} Oct 31 21:30:25.288327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.287853 4294 scope.go:122] "RemoveContainer" containerID="db22a68a16414bec7f6c39995e6b8470c46c0befe0d3491411b9fdedd927d748" Oct 31 21:30:25.289249 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.289217 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" event={"ID":"6bab82df-2cbd-401a-8095-a38ca60b9ebc","Type":"ContainerDied","Data":"1009319cedcf7c17974c04405a68e406adbd573cb04510519b4da012f5d326bb"} Oct 31 21:30:25.289342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.289245 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z" Oct 31 21:30:25.298684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.298410 4294 scope.go:122] "RemoveContainer" containerID="06f50476adb868ad2d092110e5bbcbe15c06eb105bd58ed2c5d13d624d29f830" Oct 31 21:30:25.329755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.329723 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Oct 31 21:30:25.329904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.329792 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Oct 31 21:30:25.359643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.359616 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp"] Oct 31 21:30:25.367867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.367842 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891370-j4rmp"] Oct 31 21:30:25.405227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.405199 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z"] Oct 31 21:30:25.408694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.408668 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891370-g8n8z"] Oct 31 21:30:25.987662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.987621 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" path="/var/lib/kubelet/pods/4ed8c7c1-b867-42a6-8cd7-d2c0362c0336/volumes" Oct 31 21:30:25.988245 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:25.988220 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" path="/var/lib/kubelet/pods/6bab82df-2cbd-401a-8095-a38ca60b9ebc/volumes" Oct 31 21:30:26.333892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:26.333815 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:26.334268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:26.334094 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:26.334681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:26.334666 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:30:27.628931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.628901 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t"] Oct 31 21:30:27.629343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629296 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:27.629343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629329 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:27.629343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629341 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:27.629466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629347 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:27.629466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629356 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:27.629466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629361 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:27.629466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629375 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:27.629466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629382 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:27.629466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629455 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:27.629466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629465 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:27.629682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629471 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:27.629682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.629478 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:27.633869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.633844 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.662345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.662317 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t"] Oct 31 21:30:27.675538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.675507 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.675664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.675554 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.675664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.675642 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.675794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.675776 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.675856 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.675811 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.675937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.675921 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.675990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.675967 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k65x\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-kube-api-access-2k65x\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.676042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.676020 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.676153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.676134 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.777794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.777763 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.777982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.777810 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k65x\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-kube-api-access-2k65x\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.777982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.777840 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.777982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.777877 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.777982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.777951 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.777982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.777978 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778087 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778198 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778226 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778327 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778598 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778632 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778663 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.778903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.778882 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.779996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.779975 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.780376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.780357 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.802775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.802748 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.803060 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.802999 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k65x\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-kube-api-access-2k65x\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:27.948200 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:27.948121 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:28.271857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:28.271829 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t"] Oct 31 21:30:28.273607 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:30:28.273584 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89cdf40_1bb8_4ca0_b66b_2fc6178737f9.slice/crio-4236379054d06578e541aae734f2ef870c8f651f5443f88326d87277038beae6 WatchSource:0}: Error finding container 4236379054d06578e541aae734f2ef870c8f651f5443f88326d87277038beae6: Status 404 returned error can't find the container with id 4236379054d06578e541aae734f2ef870c8f651f5443f88326d87277038beae6 Oct 31 21:30:28.276219 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:28.276189 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:28.276293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:28.276253 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:28.276293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:28.276281 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:28.318374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:28.318340 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" event={"ID":"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9","Type":"ContainerStarted","Data":"4236379054d06578e541aae734f2ef870c8f651f5443f88326d87277038beae6"} Oct 31 21:30:29.326733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:29.326692 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" event={"ID":"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9","Type":"ContainerStarted","Data":"a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e"} Oct 31 21:30:29.404362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:29.404318 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podStartSLOduration=2.404289829 podStartE2EDuration="2.404289829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:30:29.392121294 +0000 UTC m=+869.517228526" watchObservedRunningTime="2026-10-31 21:30:29.404289829 +0000 UTC m=+869.529397059" Oct 31 21:30:29.948659 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:29.948624 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:29.950231 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:29.950203 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 21:30:29.950386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:29.950261 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 21:30:30.948482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:30.948442 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 21:30:30.948869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:30.948524 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 21:30:31.333852 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:30:31.333812 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:30:31.948937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:31.948897 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 21:30:31.949320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:31.948955 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 21:30:32.949326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:32.948556 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 21:30:32.949326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:32.948626 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 21:30:33.951690 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:33.951650 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 21:30:33.952285 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:33.952243 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 21:30:34.955671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:34.955631 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Oct 31 21:30:34.956995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:34.956958 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Oct 31 21:30:36.102284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:36.102257 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:36.103033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:36.102644 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:36.103379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:36.103361 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 21:30:46.180197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:46.180162 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8"] Oct 31 21:30:58.905021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.904985 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f"] Oct 31 21:30:58.905459 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.905446 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:58.905504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.905460 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:58.905504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.905470 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:58.905504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.905475 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:58.905599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.905555 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bab82df-2cbd-401a-8095-a38ca60b9ebc" containerName="cleanup" Oct 31 21:30:58.905631 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.905608 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ed8c7c1-b867-42a6-8cd7-d2c0362c0336" containerName="cleanup" Oct 31 21:30:58.909022 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.909001 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.931666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.931636 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.931786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.931685 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.931786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.931703 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec388004-374e-4c05-b088-c430a16326ad-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.931898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.931793 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.931898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.931827 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec388004-374e-4c05-b088-c430a16326ad-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.931996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.931946 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.931996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.931986 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.932089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.932065 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.932137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.932110 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlrn6\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-kube-api-access-qlrn6\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:58.940172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:58.940143 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f"] Oct 31 21:30:59.032781 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.032754 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.032938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.032792 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec388004-374e-4c05-b088-c430a16326ad-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.032938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.032869 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.032938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.032906 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.032938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.032934 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033174 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.032955 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlrn6\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-kube-api-access-qlrn6\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033174 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033086 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033174 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033139 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033170 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec388004-374e-4c05-b088-c430a16326ad-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033334 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033377 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033575 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033620 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec388004-374e-4c05-b088-c430a16326ad-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.033719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.033645 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.035522 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.035498 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.035648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.035631 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec388004-374e-4c05-b088-c430a16326ad-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.055729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.055707 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlrn6\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-kube-api-access-qlrn6\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.055844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.055735 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.227554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.227494 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:30:59.540038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.540012 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f"] Oct 31 21:30:59.540761 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:30:59.540734 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec388004_374e_4c05_b088_c430a16326ad.slice/crio-6a0249688b83039fc1e7911132383effc882c440ce307382fa426b42baaeb88c WatchSource:0}: Error finding container 6a0249688b83039fc1e7911132383effc882c440ce307382fa426b42baaeb88c: Status 404 returned error can't find the container with id 6a0249688b83039fc1e7911132383effc882c440ce307382fa426b42baaeb88c Oct 31 21:30:59.542790 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.542762 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:59.542851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.542825 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:30:59.542887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:30:59.542853 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:00.046515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:00.046488 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:31:00.046892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:00.046637 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:31:00.541955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:00.541913 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" event={"ID":"ec388004-374e-4c05-b088-c430a16326ad","Type":"ContainerStarted","Data":"90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1"} Oct 31 21:31:00.542246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:00.542224 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" event={"ID":"ec388004-374e-4c05-b088-c430a16326ad","Type":"ContainerStarted","Data":"6a0249688b83039fc1e7911132383effc882c440ce307382fa426b42baaeb88c"} Oct 31 21:31:00.607854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:00.607801 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podStartSLOduration=2.607788315 podStartE2EDuration="2.607788315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:00.606724269 +0000 UTC m=+900.731831511" watchObservedRunningTime="2026-10-31 21:31:00.607788315 +0000 UTC m=+900.732895545" Oct 31 21:31:01.206083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.206033 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" podUID="d383ab3b-e37e-4bdf-9e81-36b8ee16070e" containerName="main" containerID="cri-o://4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854" gracePeriod=45 Oct 31 21:31:01.228545 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.228519 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:31:01.229695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.229672 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 21:31:01.229795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.229727 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 21:31:01.326713 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:31:01.326677 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:31:01.504653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.504627 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:31:01.549274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.549244 4294 generic.go:356] "Generic (PLEG): container finished" podID="d383ab3b-e37e-4bdf-9e81-36b8ee16070e" containerID="4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854" exitCode=0 Oct 31 21:31:01.549461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.549296 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" Oct 31 21:31:01.549461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.549287 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" event={"ID":"d383ab3b-e37e-4bdf-9e81-36b8ee16070e","Type":"ContainerDied","Data":"4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854"} Oct 31 21:31:01.549461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.549360 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8" event={"ID":"d383ab3b-e37e-4bdf-9e81-36b8ee16070e","Type":"ContainerDied","Data":"15aead26984c4efb8edf243d7c9dea4b6313d7eb46b12e6c6c5b0bfa5b236a23"} Oct 31 21:31:01.549461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.549384 4294 scope.go:122] "RemoveContainer" containerID="4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854" Oct 31 21:31:01.559951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.559925 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-home\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-home\") pod \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " Oct 31 21:31:01.560078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.559980 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tmp-dir\") pod \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " Oct 31 21:31:01.560484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560246 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-home" pod "d383ab3b-e37e-4bdf-9e81-36b8ee16070e" (UID: "d383ab3b-e37e-4bdf-9e81-36b8ee16070e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:31:01.560484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560257 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tmp-dir" pod "d383ab3b-e37e-4bdf-9e81-36b8ee16070e" (UID: "d383ab3b-e37e-4bdf-9e81-36b8ee16070e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:31:01.560484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560366 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-model-cache\") pod \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " Oct 31 21:31:01.560484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560413 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-kube-api-access-msrl4\" (UniqueName: \"kubernetes.io/projected/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-kube-api-access-msrl4\") pod \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " Oct 31 21:31:01.560484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560462 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-dshm\") pod \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " Oct 31 21:31:01.560696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560523 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tls-certs\") pod \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\" (UID: \"d383ab3b-e37e-4bdf-9e81-36b8ee16070e\") " Oct 31 21:31:01.560959 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560944 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.560996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.560966 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.561763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.561709 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-model-cache" pod "d383ab3b-e37e-4bdf-9e81-36b8ee16070e" (UID: "d383ab3b-e37e-4bdf-9e81-36b8ee16070e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:31:01.562745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.562726 4294 scope.go:122] "RemoveContainer" containerID="4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854" Oct 31 21:31:01.563062 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:31:01.563033 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854\": container with ID starting with 4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854 not found: ID does not exist" containerID="4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854" Oct 31 21:31:01.563150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.563076 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854"} err="failed to get container status \"4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854\": rpc error: code = NotFound desc = could not find container \"4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854\": container with ID starting with 4cfb67760588e4d7511b589f246ace2c41d1c47a6dd3762cb91812fb3bf3b854 not found: ID does not exist" Oct 31 21:31:01.563268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.563247 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tls-certs" pod "d383ab3b-e37e-4bdf-9e81-36b8ee16070e" (UID: "d383ab3b-e37e-4bdf-9e81-36b8ee16070e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:31:01.563891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.563864 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-dshm" pod "d383ab3b-e37e-4bdf-9e81-36b8ee16070e" (UID: "d383ab3b-e37e-4bdf-9e81-36b8ee16070e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:31:01.563973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.563883 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-kube-api-access-msrl4" pod "d383ab3b-e37e-4bdf-9e81-36b8ee16070e" (UID: "d383ab3b-e37e-4bdf-9e81-36b8ee16070e"). InnerVolumeSpecName "kube-api-access-msrl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:31:01.662552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.662523 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.662552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.662552 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msrl4\" (UniqueName: \"kubernetes.io/projected/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-kube-api-access-msrl4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.662722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.662565 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.662722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.662579 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d383ab3b-e37e-4bdf-9e81-36b8ee16070e-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:31:01.907675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.907648 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8"] Oct 31 21:31:01.913073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:01.913048 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-106555/e2e-conflict-a-08e7b19c-kserve-7bc7b75556-kqwc8"] Oct 31 21:31:02.000135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:02.000106 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d383ab3b-e37e-4bdf-9e81-36b8ee16070e" path="/var/lib/kubelet/pods/d383ab3b-e37e-4bdf-9e81-36b8ee16070e/volumes" Oct 31 21:31:02.228170 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:02.228088 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 21:31:02.228170 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:02.228154 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 21:31:03.228447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.228408 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 21:31:03.228905 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.228474 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 21:31:03.754984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.754951 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md"] Oct 31 21:31:03.755508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.755490 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d383ab3b-e37e-4bdf-9e81-36b8ee16070e" containerName="main" Oct 31 21:31:03.755576 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.755509 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d383ab3b-e37e-4bdf-9e81-36b8ee16070e" containerName="main" Oct 31 21:31:03.755641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.755603 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d383ab3b-e37e-4bdf-9e81-36b8ee16070e" containerName="main" Oct 31 21:31:03.760701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.760681 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:03.785666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.785641 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-106555\"/\"distinct-w3-106555-kserve-self-signed-certs\"" Oct 31 21:31:03.799543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.799507 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md"] Oct 31 21:31:03.908324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.905903 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d13372-0bc0-4ffa-886b-d91540606cbe-tls-certs\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:03.908324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.905962 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-home\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:03.908324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.906098 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4c54\" (UniqueName: \"kubernetes.io/projected/c5d13372-0bc0-4ffa-886b-d91540606cbe-kube-api-access-g4c54\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:03.908324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.906151 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-dshm\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:03.908324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.906174 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-tmp-dir\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:03.908324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:03.906210 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-model-cache\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.007186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.007081 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-dshm\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.007872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.007848 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-tmp-dir\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.008414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.008398 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-model-cache\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.008831 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.008813 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d13372-0bc0-4ffa-886b-d91540606cbe-tls-certs\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.009379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.009360 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-home\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.013578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.010045 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4c54\" (UniqueName: \"kubernetes.io/projected/c5d13372-0bc0-4ffa-886b-d91540606cbe-kube-api-access-g4c54\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.013578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.009733 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-home\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.013578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.008339 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-tmp-dir\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.013578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.008743 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-model-cache\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.023916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.023862 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d13372-0bc0-4ffa-886b-d91540606cbe-tls-certs\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.024228 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.024208 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-dshm\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.037515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.037493 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4c54\" (UniqueName: \"kubernetes.io/projected/c5d13372-0bc0-4ffa-886b-d91540606cbe-kube-api-access-g4c54\") pod \"distinct-w3-106555-kserve-84f584fddd-758md\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.116896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.113380 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:04.228061 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.228029 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 21:31:04.228171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.228100 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 21:31:04.448181 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.444746 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md"] Oct 31 21:31:04.567769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.567732 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" event={"ID":"c5d13372-0bc0-4ffa-886b-d91540606cbe","Type":"ContainerStarted","Data":"fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a"} Oct 31 21:31:04.567899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.567787 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" event={"ID":"c5d13372-0bc0-4ffa-886b-d91540606cbe","Type":"ContainerStarted","Data":"3fcfbcdf21ab8a6bd840ea28666bc7c89ff8b2db620b85a9db33ab82b1ae3cec"} Oct 31 21:31:04.618607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:04.618558 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podStartSLOduration=1.618543089 podStartE2EDuration="1.618543089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:04.616905941 +0000 UTC m=+904.742013163" watchObservedRunningTime="2026-10-31 21:31:04.618543089 +0000 UTC m=+904.743650314" Oct 31 21:31:05.114720 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:05.114693 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:05.228144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:05.228111 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 31 21:31:05.228329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:05.228173 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 31 21:31:06.450662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:06.450568 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:31:06.451431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:06.451407 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:31:06.452037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:06.452016 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 21:31:12.757116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.757082 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz"] Oct 31 21:31:12.760960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.760940 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.794134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.794104 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz"] Oct 31 21:31:12.852460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852426 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852482 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd4wn\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-kube-api-access-bd4wn\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852626 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852673 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852712 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852759 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852826 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852793 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852873 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852856 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.852914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.852895 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.953648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953615 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.953807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953661 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.953807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953682 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.953807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953705 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.953807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953730 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.953807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953750 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.953807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953805 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.954105 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953861 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd4wn\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-kube-api-access-bd4wn\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.954105 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.953946 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.954285 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.954231 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.954389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.954289 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.954518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.954490 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.954562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.954539 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.954593 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.954555 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.955835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.955817 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.956469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.956443 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.976931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.976903 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:12.977125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:12.977110 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd4wn\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-kube-api-access-bd4wn\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:13.075504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.075436 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:13.402097 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:31:13.402062 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabebb1e3_0deb_4ca7_abdc_0c609e75fbed.slice/crio-c2350d2d16c4586cdbfbf55506262a19fb16b188de0885d7dbf6fdc7e1c90470 WatchSource:0}: Error finding container c2350d2d16c4586cdbfbf55506262a19fb16b188de0885d7dbf6fdc7e1c90470: Status 404 returned error can't find the container with id c2350d2d16c4586cdbfbf55506262a19fb16b188de0885d7dbf6fdc7e1c90470 Oct 31 21:31:13.402730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.402535 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz"] Oct 31 21:31:13.405860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.405807 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:13.405969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.405939 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:13.406045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.406025 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:13.662595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.662501 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" event={"ID":"abebb1e3-0deb-4ca7-abdc-0c609e75fbed","Type":"ContainerStarted","Data":"e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384"} Oct 31 21:31:13.662595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.662562 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" event={"ID":"abebb1e3-0deb-4ca7-abdc-0c609e75fbed","Type":"ContainerStarted","Data":"c2350d2d16c4586cdbfbf55506262a19fb16b188de0885d7dbf6fdc7e1c90470"} Oct 31 21:31:13.715130 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:13.715060 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podStartSLOduration=1.715042674 podStartE2EDuration="1.715042674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:13.711761159 +0000 UTC m=+913.836868393" watchObservedRunningTime="2026-10-31 21:31:13.715042674 +0000 UTC m=+913.840149907" Oct 31 21:31:14.078321 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:14.077356 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:14.079688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:14.079382 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 21:31:14.079688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:14.079440 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 21:31:14.117557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:14.117519 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:14.131430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:14.131404 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:14.702924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:14.702890 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:31:15.096474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:15.096438 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 21:31:15.096971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:15.096515 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 21:31:16.080651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:16.080615 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 21:31:16.085422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:16.085382 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 21:31:17.076325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:17.075793 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 21:31:17.076325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:17.075861 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 21:31:18.077337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:18.076588 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 21:31:18.077337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:18.076646 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 21:31:19.076326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:19.076037 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 31 21:31:19.076326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:19.076139 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 31 21:31:20.080065 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:20.080035 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:20.080562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:20.080404 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:20.081180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:20.081158 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:31:26.409256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:26.409193 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:31:26.409895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:26.409846 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:31:31.309630 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:31:31.309594 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:31:31.568185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.568111 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf"] Oct 31 21:31:31.572111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.572090 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.594415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.594385 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf"] Oct 31 21:31:31.602782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.602760 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"distinct-w4-cc45a7-kserve-self-signed-certs\"" Oct 31 21:31:31.603159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.603119 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"openshift-service-ca.crt\"" Oct 31 21:31:31.603159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.603127 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"default-dockercfg-v4bmc\"" Oct 31 21:31:31.603159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.603127 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-cc45a7\"/\"kube-root-ca.crt\"" Oct 31 21:31:31.685686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.685659 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-dshm\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.685847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.685709 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tls-certs\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.685847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.685835 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-home\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.685964 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.685867 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-model-cache\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.686021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.685958 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kq6\" (UniqueName: \"kubernetes.io/projected/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-kube-api-access-k8kq6\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.686021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.685981 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tmp-dir\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.791031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.790984 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-dshm\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.791336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.791294 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tls-certs\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.791557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.791541 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-home\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.791685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.791671 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-model-cache\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.791859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.791844 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kq6\" (UniqueName: \"kubernetes.io/projected/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-kube-api-access-k8kq6\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.791951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.791939 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tmp-dir\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.792538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.792516 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tmp-dir\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.794187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.792882 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-home\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.794334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.793407 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-model-cache\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.802873 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.802831 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tls-certs\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.821868 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.821806 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-dshm\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.823269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.823248 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kq6\" (UniqueName: \"kubernetes.io/projected/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-kube-api-access-k8kq6\") pod \"distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:31.888566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:31.888528 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:32.191648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:32.191618 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf"] Oct 31 21:31:32.192019 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:31:32.191993 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda55beff0_3d39_41d2_9cd3_98ce50abb4ae.slice/crio-1cc4199bdeda0d16743dcaf9b8f2fa6cfa33e353a9495c79f81f23120988e37c WatchSource:0}: Error finding container 1cc4199bdeda0d16743dcaf9b8f2fa6cfa33e353a9495c79f81f23120988e37c: Status 404 returned error can't find the container with id 1cc4199bdeda0d16743dcaf9b8f2fa6cfa33e353a9495c79f81f23120988e37c Oct 31 21:31:32.895560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:32.895523 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" event={"ID":"a55beff0-3d39-41d2-9cd3-98ce50abb4ae","Type":"ContainerStarted","Data":"b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00"} Oct 31 21:31:32.895933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:32.895582 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" event={"ID":"a55beff0-3d39-41d2-9cd3-98ce50abb4ae","Type":"ContainerStarted","Data":"1cc4199bdeda0d16743dcaf9b8f2fa6cfa33e353a9495c79f81f23120988e37c"} Oct 31 21:31:32.938046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:32.938002 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podStartSLOduration=1.937989214 podStartE2EDuration="1.937989214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:32.936528305 +0000 UTC m=+933.061635533" watchObservedRunningTime="2026-10-31 21:31:32.937989214 +0000 UTC m=+933.063096444" Oct 31 21:31:33.247348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.247318 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q"] Oct 31 21:31:33.251141 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.251120 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.281652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.281620 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q"] Oct 31 21:31:33.412145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412107 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412146 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412161 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-968nf\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-kube-api-access-968nf\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412317 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412403 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412510 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412589 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f865c1-e469-4ab4-8dfd-4130094d228f-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412656 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.412728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.412703 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.513931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.513855 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.513931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.513896 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.513945 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.513964 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.513982 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-968nf\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-kube-api-access-968nf\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514018 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514047 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514134 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514183 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f865c1-e469-4ab4-8dfd-4130094d228f-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514315 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514330 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514522 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514672 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.514848 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.514829 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f865c1-e469-4ab4-8dfd-4130094d228f-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.516036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.516015 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.516469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.516452 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.538614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.538596 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.538983 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.538967 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-968nf\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-kube-api-access-968nf\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.568694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.568670 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:33.887258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.887231 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q"] Oct 31 21:31:33.888092 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:31:33.888069 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f865c1_e469_4ab4_8dfd_4130094d228f.slice/crio-4bd79aa0d3e3c7137e651cd14dc517ac1e63019b70b1a9f88d2fc84e9872ecda WatchSource:0}: Error finding container 4bd79aa0d3e3c7137e651cd14dc517ac1e63019b70b1a9f88d2fc84e9872ecda: Status 404 returned error can't find the container with id 4bd79aa0d3e3c7137e651cd14dc517ac1e63019b70b1a9f88d2fc84e9872ecda Oct 31 21:31:33.889185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.889167 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:33.890957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.890928 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:33.891051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.890988 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:33.891051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.891015 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:33.906283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:33.906255 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" event={"ID":"e5f865c1-e469-4ab4-8dfd-4130094d228f","Type":"ContainerStarted","Data":"4bd79aa0d3e3c7137e651cd14dc517ac1e63019b70b1a9f88d2fc84e9872ecda"} Oct 31 21:31:34.913633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:34.913597 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" event={"ID":"e5f865c1-e469-4ab4-8dfd-4130094d228f","Type":"ContainerStarted","Data":"7e54144c4ad675de1dbe3a36ccfc9719df09931e1de401964bcf74e47a8bf63d"} Oct 31 21:31:34.975351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:34.972074 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podStartSLOduration=1.972056771 podStartE2EDuration="1.972056771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:34.970604526 +0000 UTC m=+935.095711759" watchObservedRunningTime="2026-10-31 21:31:34.972056771 +0000 UTC m=+935.097164003" Oct 31 21:31:35.569737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:35.569702 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:35.570941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:35.570915 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:31:35.571071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:35.570967 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 21:31:36.569339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:36.569282 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:31:36.569737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:36.569367 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 21:31:37.569294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:37.569261 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:31:37.569469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:37.569351 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 21:31:38.569236 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:38.569198 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:31:38.569431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:38.569255 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 21:31:39.570037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:39.570006 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 31 21:31:39.570512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:39.570485 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 31 21:31:40.861590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:40.861557 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:40.862769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:40.862748 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:40.863502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:40.863479 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:31:41.888999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:41.888967 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:41.901295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:41.901269 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:41.962251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:41.962231 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:31:44.011598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.008142 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48"] Oct 31 21:31:44.015215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.015167 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.033473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.033436 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48"] Oct 31 21:31:44.060720 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.059670 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.061250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061190 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.062356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061430 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.062356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061478 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.062356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061503 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5zcs\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-kube-api-access-t5zcs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.062356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061533 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.062356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061644 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae5971-48ba-41ff-bc32-4580174bd6c2-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.062356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061713 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.062356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.061791 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.162685 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.162765 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.162818 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.162916 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.162953 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.162985 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.163021 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5zcs\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-kube-api-access-t5zcs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.163049 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.163161 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae5971-48ba-41ff-bc32-4580174bd6c2-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.164444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.164009 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae5971-48ba-41ff-bc32-4580174bd6c2-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.167784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.167714 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.168334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.168271 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.168697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.168675 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.169247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.169226 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.169489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.169471 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.171465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.171443 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.215770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.215703 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5zcs\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-kube-api-access-t5zcs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.216743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.216700 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.354168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.352251 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:44.746538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.746501 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48"] Oct 31 21:31:44.759671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.758916 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:44.759671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.759007 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:44.759671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.759050 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:44.995342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.994924 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" event={"ID":"5eae5971-48ba-41ff-bc32-4580174bd6c2","Type":"ContainerStarted","Data":"a302136dccbfca3fc586f3331234449f8e7db4e6a4f669226a2ad2b4f5fa5ac4"} Oct 31 21:31:44.995342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:44.994988 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" event={"ID":"5eae5971-48ba-41ff-bc32-4580174bd6c2","Type":"ContainerStarted","Data":"d83009e7bcd63e0026600313a069d8ebf45ff773d5a5cc73bd3dec68f73733ae"} Oct 31 21:31:45.356634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:45.356566 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:45.357140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:45.356709 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:31:45.357140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:45.356770 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 21:31:46.354671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:46.354635 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:31:46.354859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:46.354696 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 21:31:47.354897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:47.354852 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:31:47.355471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:47.354917 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 21:31:48.310512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.310463 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podStartSLOduration=5.310448821 podStartE2EDuration="5.310448821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:45.066614386 +0000 UTC m=+945.191721618" watchObservedRunningTime="2026-10-31 21:31:48.310448821 +0000 UTC m=+948.435556052" Oct 31 21:31:48.312771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.312750 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf"] Oct 31 21:31:48.316616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.316597 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.338683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.338659 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf"] Oct 31 21:31:48.354876 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.354848 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:31:48.354989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.354890 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 21:31:48.400842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.400799 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.400995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.400844 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqwn\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-kube-api-access-kjqwn\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.400995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.400935 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.400995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.400979 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.401177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.401066 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.401177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.401163 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.401282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.401189 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.401366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.401346 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.401425 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.401400 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.502790 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.502757 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.502790 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.502795 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.502990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.502866 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.502990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.502882 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.502990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.502914 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.502990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.502929 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqwn\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-kube-api-access-kjqwn\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.502990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.502947 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.503268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.503002 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.503268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.503090 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.503268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.503217 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-credential-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.503268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.503236 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-socket\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.503684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.503656 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-data\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.503764 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.503699 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-certs\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.503827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.503789 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istiod-ca-cert\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.505089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.505064 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-envoy\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.505492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.505475 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-podinfo\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.525873 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.525845 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-token\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.526015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.526000 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqwn\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-kube-api-access-kjqwn\") pod \"e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:48.633564 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:48.633481 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:49.000966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:49.000936 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf"] Oct 31 21:31:49.001380 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:31:49.001168 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a24e18f_9d3c_4e4b_8eaf_9703c80cfa6f.slice/crio-035f1858bba8537c765cb3db91a0920c68390fc7a193e8ed66dfd05e4c07b46f WatchSource:0}: Error finding container 035f1858bba8537c765cb3db91a0920c68390fc7a193e8ed66dfd05e4c07b46f: Status 404 returned error can't find the container with id 035f1858bba8537c765cb3db91a0920c68390fc7a193e8ed66dfd05e4c07b46f Oct 31 21:31:49.004967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:49.004935 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:49.005052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:49.004998 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:49.005052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:49.005024 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:49.032256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:49.032206 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" event={"ID":"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f","Type":"ContainerStarted","Data":"035f1858bba8537c765cb3db91a0920c68390fc7a193e8ed66dfd05e4c07b46f"} Oct 31 21:31:49.355152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:49.355064 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:31:49.355723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:49.355684 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 21:31:50.039984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:50.039953 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" event={"ID":"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f","Type":"ContainerStarted","Data":"30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b"} Oct 31 21:31:50.130636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:50.130573 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podStartSLOduration=2.13055417 podStartE2EDuration="2.13055417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:50.114460264 +0000 UTC m=+950.239567496" watchObservedRunningTime="2026-10-31 21:31:50.13055417 +0000 UTC m=+950.255661403" Oct 31 21:31:50.354759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:50.354672 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:31:50.354759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:50.354734 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 21:31:50.633674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:50.633591 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:50.635080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:50.635052 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:31:50.635208 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:50.635104 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 21:31:51.355018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:51.354988 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 31 21:31:51.355250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:51.355230 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 31 21:31:51.634739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:51.634652 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:31:51.634739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:51.634717 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 21:31:52.634857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:52.634818 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:31:52.635270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:52.634881 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 21:31:52.891683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:52.891600 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:52.894296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:52.894273 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:52.900497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:52.900477 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 21:31:53.562376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.562345 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6"] Oct 31 21:31:53.566453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.566430 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.596324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.595622 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-994336-openshift-default-dockercfg-qgxk4\"" Oct 31 21:31:53.620584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.620550 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6"] Oct 31 21:31:53.634674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.634645 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:31:53.634826 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.634702 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 21:31:53.670681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.670651 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4b0b62-3214-4588-84f5-0e9717bf797c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.670692 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.670795 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.670830 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.670961 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671198 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.671014 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671198 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.671092 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqrn\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-kube-api-access-nzqrn\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671198 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.671150 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.671355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.671206 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.793840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.793801 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.794100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.794078 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.794284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.794268 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.794436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.794420 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.794546 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.794531 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqrn\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-kube-api-access-nzqrn\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.794712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.794695 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.794845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.794829 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.796605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.795043 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4b0b62-3214-4588-84f5-0e9717bf797c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.796605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.795421 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.796605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.796429 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.796605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.796155 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.796605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.795940 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.796953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.796749 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4b0b62-3214-4588-84f5-0e9717bf797c-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.796953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.796800 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.798910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.798866 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.799224 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.799201 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.818997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.818937 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.818997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.818955 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqrn\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-kube-api-access-nzqrn\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:53.911253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:53.911214 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:54.229628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:54.229604 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6"] Oct 31 21:31:54.231067 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:31:54.231039 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4b0b62_3214_4588_84f5_0e9717bf797c.slice/crio-b6bab8e07e687c79eb274e5e3a011f9cb7989893b8134d64149cb0dfe1631f96 WatchSource:0}: Error finding container b6bab8e07e687c79eb274e5e3a011f9cb7989893b8134d64149cb0dfe1631f96: Status 404 returned error can't find the container with id b6bab8e07e687c79eb274e5e3a011f9cb7989893b8134d64149cb0dfe1631f96 Oct 31 21:31:54.233524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:54.233492 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:54.233592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:54.233554 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:54.233592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:54.233583 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:31:54.634684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:54.634647 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:31:54.634863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:54.634707 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 21:31:55.071056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.071017 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" event={"ID":"ea4b0b62-3214-4588-84f5-0e9717bf797c","Type":"ContainerStarted","Data":"2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297"} Oct 31 21:31:55.084529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.071075 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" event={"ID":"ea4b0b62-3214-4588-84f5-0e9717bf797c","Type":"ContainerStarted","Data":"b6bab8e07e687c79eb274e5e3a011f9cb7989893b8134d64149cb0dfe1631f96"} Oct 31 21:31:55.132788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.132735 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" podStartSLOduration=2.132700078 podStartE2EDuration="2.132700078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:31:55.11938248 +0000 UTC m=+955.244489710" watchObservedRunningTime="2026-10-31 21:31:55.132700078 +0000 UTC m=+955.257807308" Oct 31 21:31:55.634479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.634447 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 31 21:31:55.634648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.634498 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 31 21:31:55.912246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.912162 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:31:55.913704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.913678 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:31:55.913864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:55.913734 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 21:31:56.638378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:56.638343 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:56.638910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:56.638893 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:56.639739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:56.639721 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:31:56.911874 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:56.911793 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:31:56.912113 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:56.912087 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 21:31:57.911949 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:57.911914 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:31:57.912446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:57.911981 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 21:31:58.911554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:58.911522 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:31:58.911722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:58.911572 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 21:31:59.912005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:59.911966 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 21:31:59.912507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:31:59.912030 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 21:32:01.085847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:01.085786 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:32:01.087058 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:01.087038 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:32:01.088498 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:01.088475 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 21:32:01.314372 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:32:01.314333 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:32:05.674456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:05.674408 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:06.789240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:06.789204 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:06.789833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:06.789798 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:06.891446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:06.891386 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:06.969597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:06.969493 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 21:32:07.083535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.083429 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:07.157442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.157393 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:07.290470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.290432 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:07.290766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.290721 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:07.382085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.381456 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:07.614747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.614402 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:07.614747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.614468 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:07.868945 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.868400 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:07.898223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.898181 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:07.898887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.898857 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:32:07.930412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:07.929959 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:32:24.938329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:24.934483 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:24.961337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:24.959395 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.073468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:25.073416 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.074123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:25.073736 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 21:32:25.117691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:25.117388 4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.131340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:25.127391 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.277759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:25.277702 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:25.684862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:25.684752 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:26.042340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.015557 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 21:32:26.292328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.232942 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 21:32:26.292328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.234498 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 21:32:26.337434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.316587 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:26.419193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.419155 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:32:26.458026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.457992 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:32:26.458378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.458357 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:32:26.458657 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.458642 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:32:26.728386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.726497 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 21:32:26.728386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.726839 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:32:26.909326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.900706 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 21:32:26.945793 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.941813 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:26.971331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.949487 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:26.995382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:26.975861 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:32:27.016193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.016124 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:27.045435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.043515 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 21:32:27.236258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.235424 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 21:32:27.236258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.235748 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 21:32:27.320497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.320387 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:27.373580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.372546 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:32:27.425551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.424939 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:32:27.425551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.424994 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:27.861466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.854404 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:32:27.882401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.882357 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:32:27.893327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:27.885116 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:32:31.312316 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:32:31.312265 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:32:46.862323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:46.861434 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:46.872339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:46.868927 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 21:32:46.878398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:46.876862 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:46.878398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:46.877633 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:46.939320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:46.938503 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:47.000340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:46.994743 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:47.148382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.127457 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 21:32:47.311676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.311624 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:47.333664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.333619 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.073677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.867022 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.073677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.867485 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:32:48.073677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.955817 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.073677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.956676 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:32:48.073677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.957878 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.073677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.958204 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:32:48.073677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.069431 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 21:32:48.128401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:47.940296 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.192131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.178327 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:32:48.216680 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.216650 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:32:48.387323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.337361 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.389477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.337395 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 21:32:48.438326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.435836 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:32:48.438326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.436144 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:32:48.513658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.513629 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:32:48.956454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.956397 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.973991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.956693 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:48.975343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:48.974793 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.072846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.072796 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.201958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.197700 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.201958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.198092 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded" Oct 31 21:32:49.438588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.437238 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.578323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.570249 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.639732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.639696 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww"] Oct 31 21:32:49.687996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.687960 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.697376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.697276 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww"] Oct 31 21:32:49.810149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.809287 4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847202 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847274 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847322 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847382 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847459 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847606 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847677 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6kzx\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-kube-api-access-v6kzx\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847736 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.850431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.847760 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.945361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.944932 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.949719 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.949774 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.949830 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6kzx\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-kube-api-access-v6kzx\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.949885 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.949909 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.950024 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.950069 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.950098 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.950155 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.950629 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.951189 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.951725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.951699 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.952073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.952054 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.952406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.952381 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.953273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.952710 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:32:49.962112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.961557 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.967559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.967535 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.985152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.985124 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:49.986487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.986451 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:32:49.988402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:49.988356 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6kzx\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-kube-api-access-v6kzx\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:50.156329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.156274 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:50.169331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.169137 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:32:50.169331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.169203 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:32:50.169331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.169240 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:32:50.169331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.169275 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:32:50.343267 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.343236 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:50.697188 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:32:50.697155 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e16965_03a7_4bc3_a07d_0ed3761cb62d.slice/crio-751ebf31bb8d7110cb8f80f90af3fda26e3640838a2fc543ce8e72b82a1523b6 WatchSource:0}: Error finding container 751ebf31bb8d7110cb8f80f90af3fda26e3640838a2fc543ce8e72b82a1523b6: Status 404 returned error can't find the container with id 751ebf31bb8d7110cb8f80f90af3fda26e3640838a2fc543ce8e72b82a1523b6 Oct 31 21:32:50.697362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.697345 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww"] Oct 31 21:32:50.699543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.699505 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:32:50.699639 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.699575 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:32:50.699639 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:50.699606 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:32:51.409038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:51.409002 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" event={"ID":"10e16965-03a7-4bc3-a07d-0ed3761cb62d","Type":"ContainerStarted","Data":"39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff"} Oct 31 21:32:51.409470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:51.409060 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" event={"ID":"10e16965-03a7-4bc3-a07d-0ed3761cb62d","Type":"ContainerStarted","Data":"751ebf31bb8d7110cb8f80f90af3fda26e3640838a2fc543ce8e72b82a1523b6"} Oct 31 21:32:51.487842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:51.487781 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podStartSLOduration=2.487761648 podStartE2EDuration="2.487761648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:51.470689058 +0000 UTC m=+1011.595796323" watchObservedRunningTime="2026-10-31 21:32:51.487761648 +0000 UTC m=+1011.612868881" Oct 31 21:32:52.343607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:52.343567 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:52.344747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:52.344722 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 21:32:52.344868 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:52.344768 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 21:32:53.343821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.343787 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 21:32:53.344351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.343840 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 21:32:53.801140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.801098 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5"] Oct 31 21:32:53.805249 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.805216 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.835814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.835774 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5"] Oct 31 21:32:53.930116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930069 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930138 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930251 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930281 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930377 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930393 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmxb\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-kube-api-access-4vmxb\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930518 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81bf27-72f7-4a89-8908-49f0c75c0306-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930641 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:53.930739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:53.930681 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.031981 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032144 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032221 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032190 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032274 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032364 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmxb\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-kube-api-access-4vmxb\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032425 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032412 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032428 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81bf27-72f7-4a89-8908-49f0c75c0306-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032538 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032564 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032573 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032654 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032882 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.032988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.032948 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.033066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.033047 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81bf27-72f7-4a89-8908-49f0c75c0306-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.034555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.034532 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.034890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.034869 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.068803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.068726 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.068976 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.068954 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmxb\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-kube-api-access-4vmxb\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.123916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.122558 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:54.344797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.344705 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 21:32:54.344797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.344768 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 21:32:54.514207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.514177 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5"] Oct 31 21:32:54.516613 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:32:54.516571 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e81bf27_72f7_4a89_8908_49f0c75c0306.slice/crio-0baf2ca7a7f0a33c8b88fcc2470217b82b96c8e4211edee55f81e288ef5d8da8 WatchSource:0}: Error finding container 0baf2ca7a7f0a33c8b88fcc2470217b82b96c8e4211edee55f81e288ef5d8da8: Status 404 returned error can't find the container with id 0baf2ca7a7f0a33c8b88fcc2470217b82b96c8e4211edee55f81e288ef5d8da8 Oct 31 21:32:54.518825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.518792 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:32:54.518904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.518861 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:32:54.518904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:54.518893 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:32:55.344403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:55.344366 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 21:32:55.344608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:55.344424 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 21:32:55.439647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:55.439610 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" event={"ID":"8e81bf27-72f7-4a89-8908-49f0c75c0306","Type":"ContainerStarted","Data":"cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa"} Oct 31 21:32:55.440034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:55.439668 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" event={"ID":"8e81bf27-72f7-4a89-8908-49f0c75c0306","Type":"ContainerStarted","Data":"0baf2ca7a7f0a33c8b88fcc2470217b82b96c8e4211edee55f81e288ef5d8da8"} Oct 31 21:32:55.513327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:55.503122 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podStartSLOduration=2.503094985 podStartE2EDuration="2.503094985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:32:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:32:55.500960422 +0000 UTC m=+1015.626067652" watchObservedRunningTime="2026-10-31 21:32:55.503094985 +0000 UTC m=+1015.628202216" Oct 31 21:32:56.124548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:56.124345 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:32:56.126095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:56.126066 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 21:32:56.126227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:56.126123 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 21:32:56.344560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:56.344524 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 21:32:56.344751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:56.344582 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 21:32:57.124518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:57.124482 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 21:32:57.124999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:57.124534 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 21:32:57.343925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:57.343889 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 21:32:57.344107 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:57.343943 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 21:32:58.124094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:58.124057 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 21:32:58.124292 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:58.124116 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 21:32:58.349086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:58.349053 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:58.349525 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:58.349503 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:58.350117 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:58.350100 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:32:59.124148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:59.124111 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 21:32:59.124392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:32:59.124173 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 21:33:00.124029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:00.123993 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 21:33:00.124453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:00.124051 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 21:33:01.124161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:01.124128 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 31 21:33:01.124652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:01.124196 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 31 21:33:01.321925 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:33:01.321859 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:33:02.128079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:02.128047 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:33:02.128562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:02.128392 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:33:02.129123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:02.129105 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:33:08.843156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.843108 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw"] Oct 31 21:33:08.847625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.847608 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.884193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.884148 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw"] Oct 31 21:33:08.964475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.964433 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.964824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.964798 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh7hx\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-kube-api-access-kh7hx\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.964955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.964938 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.965164 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.965147 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.965389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.965372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.965596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.965577 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.965764 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.965748 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.965879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.965864 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:08.965994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:08.965978 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.068155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.068111 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.069989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.069928 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.070260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.070239 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.070527 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.070506 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.070956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.070940 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.071432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.071413 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.071882 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.071866 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.072535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.072516 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh7hx\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-kube-api-access-kh7hx\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.072953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.072935 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.083479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.070896 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.084095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.073284 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.084236 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.071780 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.084393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.083719 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.084648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.084014 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.091821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.091792 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.102841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.102777 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.104070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.104043 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh7hx\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-kube-api-access-kh7hx\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.122133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.122099 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:09.161277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:09.161245 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:10.573794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.573483 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:10.578406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.575673 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 21:33:10.627756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.627702 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:10.922383 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.921887 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw"] Oct 31 21:33:10.955325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.950323 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:33:10.955325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.950428 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:33:10.955325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.950462 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:33:10.955325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.951041 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:10.997329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:10.992487 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.110232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.108412 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" event={"ID":"93ba5bd9-d03d-4399-a8b7-173694f0a66f","Type":"ContainerStarted","Data":"1b832deabdcd345cf2d4404bf741537c4e88f41f598aad0941f7fcd58d9ac776"} Oct 31 21:33:11.246159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.232446 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.254331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.248569 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.270649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.270604 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.582384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.573504 4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded" Oct 31 21:33:11.582384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.573830 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.583525 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.583461 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:33:11.585100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.585068 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.585468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.585438 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:33:11.642327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.635647 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.642327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.636473 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:33:11.936485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.936382 4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.963091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.962731 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:11.972327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.964112 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:33:12.000610 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:11.999605 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.086125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.086086 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:33:12.175425 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.175378 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.243329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.237672 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.243329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.238362 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:33:12.260739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.260669 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.264511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.264474 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:33:12.294938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.292825 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.294938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.293213 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:33:12.361425 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.361386 4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.570284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.570197 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" event={"ID":"93ba5bd9-d03d-4399-a8b7-173694f0a66f","Type":"ContainerStarted","Data":"4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952"} Oct 31 21:33:12.582446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.582401 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.675203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.649423 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:12.675203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.649811 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded" Oct 31 21:33:12.874725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.865927 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podStartSLOduration=4.8659057919999995 podStartE2EDuration="4.865905792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:12.678653151 +0000 UTC m=+1032.803760383" watchObservedRunningTime="2026-10-31 21:33:12.865905792 +0000 UTC m=+1032.991013024" Oct 31 21:33:12.970348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:12.963528 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded" Oct 31 21:33:13.055361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.024445 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:13.140930 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.140839 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 21:33:13.192346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.142281 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 21:33:13.207474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.207440 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:33:13.207750 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.207715 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 21:33:13.208897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.208858 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:13.279084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.279006 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 31 21:33:13.285478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.285428 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:13.301440 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.293392 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:13.634382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.612882 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:13.658515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.658474 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:13.659519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.659489 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 21:33:13.799362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.795063 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 21:33:13.818709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.818671 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 21:33:13.818928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.818909 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 21:33:13.819233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.819212 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 21:33:13.821253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.820791 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 21:33:13.838232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:13.835706 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 21:33:14.162110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:14.162076 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:33:14.162404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:14.162359 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 21:33:15.162472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:15.162437 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:33:15.162884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:15.162494 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 21:33:16.161780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:16.161744 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:33:16.161956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:16.161793 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 21:33:17.162480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:17.162450 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:33:17.162860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:17.162507 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 21:33:18.161914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:18.161874 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:33:18.162099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:18.161929 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 21:33:19.162748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:19.162712 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 31 21:33:19.163138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:19.162778 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 31 21:33:20.166373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:20.166343 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:20.166847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:20.166618 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:20.167353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:20.167297 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:33:30.616624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:30.616584 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:33:30.661134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:30.661092 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 21:33:31.316223 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:33:31.316144 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:33:33.593410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.593377 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt"] Oct 31 21:33:33.597605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.597587 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.618998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.618971 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt"] Oct 31 21:33:33.627951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.627924 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-084d807f-serving-cert\"" Oct 31 21:33:33.703772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.703737 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.703931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.703879 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8dj2\" (UniqueName: \"kubernetes.io/projected/d033fb75-e793-442f-a7d8-fc9bcccb1495-kube-api-access-s8dj2\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.704004 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.703991 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-metrics-certs\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.704040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.704023 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d033fb75-e793-442f-a7d8-fc9bcccb1495-openshift-service-ca\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.805581 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.805541 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dj2\" (UniqueName: \"kubernetes.io/projected/d033fb75-e793-442f-a7d8-fc9bcccb1495-kube-api-access-s8dj2\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.805795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.805757 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-metrics-certs\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.805862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.805806 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d033fb75-e793-442f-a7d8-fc9bcccb1495-openshift-service-ca\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.805922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.805893 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.806138 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:33:33.806119 4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-serving-cert: secret "maas-api-e2e-mt-084d807f-serving-cert" not found Oct 31 21:33:33.806242 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:33:33.806207 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls podName:d033fb75-e793-442f-a7d8-fc9bcccb1495 nodeName:}" failed. No retries permitted until 2026-10-31 21:33:34.306183947 +0000 UTC m=+1054.431291183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls") pod "maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" (UID: "d033fb75-e793-442f-a7d8-fc9bcccb1495") : secret "maas-api-e2e-mt-084d807f-serving-cert" not found Oct 31 21:33:33.808116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.808077 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d033fb75-e793-442f-a7d8-fc9bcccb1495-openshift-service-ca\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.810265 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.810217 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-metrics-certs\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:33.839082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:33.839007 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8dj2\" (UniqueName: \"kubernetes.io/projected/d033fb75-e793-442f-a7d8-fc9bcccb1495-kube-api-access-s8dj2\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:34.312206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:34.312172 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:34.314705 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:34.314674 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:34.510543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:34.510507 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:34.830778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:34.830755 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt"] Oct 31 21:33:34.833045 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:33:34.833018 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd033fb75_e793_442f_a7d8_fc9bcccb1495.slice/crio-a290850a378a782d87c9b6eea89cc34ba4aad2c213bdd65d4a646c7f0ffd5cd4 WatchSource:0}: Error finding container a290850a378a782d87c9b6eea89cc34ba4aad2c213bdd65d4a646c7f0ffd5cd4: Status 404 returned error can't find the container with id a290850a378a782d87c9b6eea89cc34ba4aad2c213bdd65d4a646c7f0ffd5cd4 Oct 31 21:33:34.834535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:34.834516 4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:33:34.924333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:34.919844 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" event={"ID":"d033fb75-e793-442f-a7d8-fc9bcccb1495","Type":"ContainerStarted","Data":"a290850a378a782d87c9b6eea89cc34ba4aad2c213bdd65d4a646c7f0ffd5cd4"} Oct 31 21:33:35.933097 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:35.933053 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" event={"ID":"d033fb75-e793-442f-a7d8-fc9bcccb1495","Type":"ContainerStarted","Data":"78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29"} Oct 31 21:33:35.933782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:35.933759 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:35.978315 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:35.978243 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" podStartSLOduration=2.5395851179999998 podStartE2EDuration="2.978225628s" totalImagesPullingTime="438.64051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:35.975945751 +0000 UTC m=+1056.101052981" watchObservedRunningTime="2026-10-31 21:33:35.978225628 +0000 UTC m=+1056.103332860" Oct 31 21:33:41.943669 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:41.943634 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:33:49.598992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.598952 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77"] Oct 31 21:33:49.603746 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.603709 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.631578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.631549 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77"] Oct 31 21:33:49.680103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680056 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680128 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680245 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680355 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680418 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680543 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680638 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680793 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.680939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.680916 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4mmb\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-kube-api-access-s4mmb\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.782856 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.782806 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.782887 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.782941 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.782965 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783081 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783201 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4mmb\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-kube-api-access-s4mmb\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783250 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783283 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783321 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783365 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783431 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783606 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783574 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783709 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.783819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.783797 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.785410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.785388 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.785587 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.785565 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.806631 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.806608 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.806857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.806834 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4mmb\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-kube-api-access-s4mmb\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.942407 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.942295 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:49.964496 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.964464 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx"] Oct 31 21:33:49.969813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.969774 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:49.990713 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:49.989157 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7ac3c8-serving-cert\"" Oct 31 21:33:50.001453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.000570 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx"] Oct 31 21:33:50.087699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.087660 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-maas-api-tls\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.087882 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.087708 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e592a26d-cc33-4400-9b83-d5a2f204a632-openshift-service-ca\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.087954 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.087940 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnpz\" (UniqueName: \"kubernetes.io/projected/e592a26d-cc33-4400-9b83-d5a2f204a632-kube-api-access-xbnpz\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.088501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.088471 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-metrics-certs\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.189951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.189917 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnpz\" (UniqueName: \"kubernetes.io/projected/e592a26d-cc33-4400-9b83-d5a2f204a632-kube-api-access-xbnpz\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.190131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.189989 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-metrics-certs\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.190207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.190127 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-maas-api-tls\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.190207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.190154 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e592a26d-cc33-4400-9b83-d5a2f204a632-openshift-service-ca\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.190814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.190789 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e592a26d-cc33-4400-9b83-d5a2f204a632-openshift-service-ca\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.192490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.192445 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-metrics-certs\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.192490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.192482 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-maas-api-tls\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.235940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.235908 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnpz\" (UniqueName: \"kubernetes.io/projected/e592a26d-cc33-4400-9b83-d5a2f204a632-kube-api-access-xbnpz\") pod \"maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.288648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.288612 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:50.301286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.301253 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77"] Oct 31 21:33:50.307841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.307809 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:33:50.307961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.307891 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:33:50.307961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.307938 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:33:50.626566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:50.626495 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx"] Oct 31 21:33:50.635670 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:33:50.635630 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode592a26d_cc33_4400_9b83_d5a2f204a632.slice/crio-b65bfaad09dd56d1688049915eb82cddbcfd91e66d00184c144afc1b3ee32a13 WatchSource:0}: Error finding container b65bfaad09dd56d1688049915eb82cddbcfd91e66d00184c144afc1b3ee32a13: Status 404 returned error can't find the container with id b65bfaad09dd56d1688049915eb82cddbcfd91e66d00184c144afc1b3ee32a13 Oct 31 21:33:51.042155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:51.042113 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" event={"ID":"e592a26d-cc33-4400-9b83-d5a2f204a632","Type":"ContainerStarted","Data":"b65bfaad09dd56d1688049915eb82cddbcfd91e66d00184c144afc1b3ee32a13"} Oct 31 21:33:51.044260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:51.044230 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" event={"ID":"cbc6f25d-3819-456d-b152-a2fe2b4a7e11","Type":"ContainerStarted","Data":"521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa"} Oct 31 21:33:51.044479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:51.044281 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" event={"ID":"cbc6f25d-3819-456d-b152-a2fe2b4a7e11","Type":"ContainerStarted","Data":"65fc50399a161e54b53fab23950b1137d1cadebcfe72ca4c356b76bb068ac348"} Oct 31 21:33:51.119752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:51.118489 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podStartSLOduration=2.118467423 podStartE2EDuration="2.118467423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:51.115746941 +0000 UTC m=+1071.240854173" watchObservedRunningTime="2026-10-31 21:33:51.118467423 +0000 UTC m=+1071.243574655" Oct 31 21:33:51.943065 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:51.943027 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:51.944631 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:51.944609 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 21:33:51.944731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:51.944653 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 21:33:52.051628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:52.051513 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" event={"ID":"e592a26d-cc33-4400-9b83-d5a2f204a632","Type":"ContainerStarted","Data":"246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc"} Oct 31 21:33:52.051628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:52.051606 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:52.103020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:52.102952 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" podStartSLOduration=2.660436805 podStartE2EDuration="3.102933509s" totalImagesPullingTime="442.496704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:52.096726623 +0000 UTC m=+1072.221833855" watchObservedRunningTime="2026-10-31 21:33:52.102933509 +0000 UTC m=+1072.228040742" Oct 31 21:33:52.943642 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:52.943606 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 21:33:52.944119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:52.943659 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 21:33:53.943244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:53.943208 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 21:33:53.943411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:53.943272 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 21:33:54.943356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:54.943284 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 21:33:54.943840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:54.943372 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 21:33:55.943262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:55.943221 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 21:33:55.943460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:55.943282 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 21:33:56.944029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:56.943979 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 21:33:56.944539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:56.944050 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 21:33:57.197661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.197561 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb"] Oct 31 21:33:57.202312 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.202274 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.221681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.221656 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb"] Oct 31 21:33:57.229513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.229481 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-371531\"" Oct 31 21:33:57.229633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.229550 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-371531-dockercfg-mpgp5\"" Oct 31 21:33:57.334968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.334923 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqhf8\" (UniqueName: \"kubernetes.io/projected/95989257-2156-4ffa-b83a-73ff9c0701a4-kube-api-access-vqhf8\") pod \"payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.335156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.335129 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95989257-2156-4ffa-b83a-73ff9c0701a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.436974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.436933 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95989257-2156-4ffa-b83a-73ff9c0701a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.437236 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.437086 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqhf8\" (UniqueName: \"kubernetes.io/projected/95989257-2156-4ffa-b83a-73ff9c0701a4-kube-api-access-vqhf8\") pod \"payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.437719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.437699 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95989257-2156-4ffa-b83a-73ff9c0701a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.462991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.462905 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqhf8\" (UniqueName: \"kubernetes.io/projected/95989257-2156-4ffa-b83a-73ff9c0701a4-kube-api-access-vqhf8\") pod \"payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.519502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.519460 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:33:57.849817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.849787 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb"] Oct 31 21:33:57.852855 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:33:57.852830 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95989257_2156_4ffa_b83a_73ff9c0701a4.slice/crio-abd5add6ae1517c207114d589a4f2a705e92a153b23e2a911994beaac01d1027 WatchSource:0}: Error finding container abd5add6ae1517c207114d589a4f2a705e92a153b23e2a911994beaac01d1027: Status 404 returned error can't find the container with id abd5add6ae1517c207114d589a4f2a705e92a153b23e2a911994beaac01d1027 Oct 31 21:33:57.943601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.943565 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 31 21:33:57.943775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.943628 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 31 21:33:57.989169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.989137 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg"] Oct 31 21:33:57.994655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:57.994633 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.007416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.007393 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg"] Oct 31 21:33:58.027921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.027889 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-084d807f\"/\"openshift-service-ca.crt\"" Oct 31 21:33:58.028090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.027959 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-084d807f\"/\"kube-root-ca.crt\"" Oct 31 21:33:58.064906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.064873 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:33:58.082879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.082845 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-084d807f\"/\"e2e-lifecycle-model-084d807f-kserve-self-signed-certs\"" Oct 31 21:33:58.083067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.082954 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-084d807f\"/\"default-dockercfg-msvqw\"" Oct 31 21:33:58.090931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.090896 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" event={"ID":"95989257-2156-4ffa-b83a-73ff9c0701a4","Type":"ContainerStarted","Data":"abd5add6ae1517c207114d589a4f2a705e92a153b23e2a911994beaac01d1027"} Oct 31 21:33:58.151375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.148041 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-dshm\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.151375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.148077 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tls-certs\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.151375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.148096 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jj4\" (UniqueName: \"kubernetes.io/projected/fbe06c24-13f3-482f-9be7-e71f02a3faa1-kube-api-access-c4jj4\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.151375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.148148 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-home\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.151375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.148393 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-model-cache\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.151375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.148438 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tmp-dir\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.253891 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-model-cache\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.253960 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tmp-dir\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.254072 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-dshm\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.254149 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tls-certs\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.254174 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jj4\" (UniqueName: \"kubernetes.io/projected/fbe06c24-13f3-482f-9be7-e71f02a3faa1-kube-api-access-c4jj4\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.254232 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-home\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.254489 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-model-cache\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.254967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.254684 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-home\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.255574 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.255066 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tmp-dir\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.287325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.284089 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-dshm\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.296353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.294929 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tls-certs\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.310327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.302342 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jj4\" (UniqueName: \"kubernetes.io/projected/fbe06c24-13f3-482f-9be7-e71f02a3faa1-kube-api-access-c4jj4\") pod \"e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.314542 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.313630 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:33:58.711134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.711101 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg"] Oct 31 21:33:58.949000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.948969 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:58.949375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.949345 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:58.950214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:58.950195 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:33:59.115153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:59.115056 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" event={"ID":"fbe06c24-13f3-482f-9be7-e71f02a3faa1","Type":"ContainerStarted","Data":"097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39"} Oct 31 21:33:59.115153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:59.115112 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" event={"ID":"fbe06c24-13f3-482f-9be7-e71f02a3faa1","Type":"ContainerStarted","Data":"8359b22cbeb11fcefdc9580c8ae2fedc02b8cfd3c50dda998a350bc8c0fe60bf"} Oct 31 21:33:59.168917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:59.168856 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" podStartSLOduration=2.168838923 podStartE2EDuration="2.168838923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:33:59.16876899 +0000 UTC m=+1079.293876219" watchObservedRunningTime="2026-10-31 21:33:59.168838923 +0000 UTC m=+1079.293946156" Oct 31 21:33:59.314071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:33:59.314037 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:34:01.129155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:01.129112 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" event={"ID":"95989257-2156-4ffa-b83a-73ff9c0701a4","Type":"ContainerStarted","Data":"64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86"} Oct 31 21:34:01.129670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:01.129330 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:34:01.175872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:01.175801 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" podStartSLOduration=1.791615485 podStartE2EDuration="4.175789105s" totalImagesPullingTime="2.38417362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:34:01.173419186 +0000 UTC m=+1081.298526418" watchObservedRunningTime="2026-10-31 21:34:01.175789105 +0000 UTC m=+1081.300896336" Oct 31 21:34:01.318038 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:34:01.317993 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:34:08.314381 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:08.314342 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:34:08.326802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:08.326772 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:34:09.189653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:09.189619 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:34:11.695262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.695227 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94"] Oct 31 21:34:11.700777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.700759 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.716230 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.716204 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-371531-serving-cert\"" Oct 31 21:34:11.733402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.733368 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94"] Oct 31 21:34:11.756252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.756165 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-openshift-service-ca\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.756472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.756269 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-metrics-certs\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.756472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.756457 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.756614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.756520 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt97z\" (UniqueName: \"kubernetes.io/projected/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-kube-api-access-bt97z\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.857932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.857893 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.858110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.857962 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt97z\" (UniqueName: \"kubernetes.io/projected/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-kube-api-access-bt97z\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.858110 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:34:11.858036 4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-371531-serving-cert: secret "maas-api-e2e-api-b-371531-serving-cert" not found Oct 31 21:34:11.858110 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:34:11.858098 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls podName:5f1567ac-8c97-4c54-a0b2-fdd6a321f435 nodeName:}" failed. No retries permitted until 2026-10-31 21:34:12.358082286 +0000 UTC m=+1092.483189533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls") pod "maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" (UID: "5f1567ac-8c97-4c54-a0b2-fdd6a321f435") : secret "maas-api-e2e-api-b-371531-serving-cert" not found Oct 31 21:34:11.858257 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.858126 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-openshift-service-ca\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.858257 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.858160 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-metrics-certs\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.858821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.858795 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-openshift-service-ca\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.861123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.861106 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-metrics-certs\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:11.882899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:11.882864 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt97z\" (UniqueName: \"kubernetes.io/projected/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-kube-api-access-bt97z\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:12.138738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:12.138712 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:34:12.364889 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:12.364854 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:12.367241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:12.367220 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls\") pod \"maas-api-e2e-api-b-371531-59df8fbf9f-hrq94\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:12.613973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:12.613927 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:12.951679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:12.951656 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94"] Oct 31 21:34:12.953031 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:34:12.953006 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1567ac_8c97_4c54_a0b2_fdd6a321f435.slice/crio-b0a3fd54aa300f1a3e45d961182e8717e6a68393d9066b77da9c3e309048600c WatchSource:0}: Error finding container b0a3fd54aa300f1a3e45d961182e8717e6a68393d9066b77da9c3e309048600c: Status 404 returned error can't find the container with id b0a3fd54aa300f1a3e45d961182e8717e6a68393d9066b77da9c3e309048600c Oct 31 21:34:13.204751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:13.204664 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" event={"ID":"5f1567ac-8c97-4c54-a0b2-fdd6a321f435","Type":"ContainerStarted","Data":"b0a3fd54aa300f1a3e45d961182e8717e6a68393d9066b77da9c3e309048600c"} Oct 31 21:34:14.212417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:14.212381 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" event={"ID":"5f1567ac-8c97-4c54-a0b2-fdd6a321f435","Type":"ContainerStarted","Data":"3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f"} Oct 31 21:34:14.212806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:14.212496 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:14.256261 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:14.256214 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" podStartSLOduration=2.659053901 podStartE2EDuration="3.256198961s" totalImagesPullingTime="597.14506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:34:14.254572827 +0000 UTC m=+1094.379680058" watchObservedRunningTime="2026-10-31 21:34:14.256198961 +0000 UTC m=+1094.381306222" Oct 31 21:34:20.225140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:20.225113 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:34:21.454126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.454092 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn"] Oct 31 21:34:21.458192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.458171 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:21.477071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.477042 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn"] Oct 31 21:34:21.485319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.485283 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-248b29-dockercfg-85g7l\"" Oct 31 21:34:21.485464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.485360 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-248b29\"" Oct 31 21:34:21.622788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.622726 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmm9x\" (UniqueName: \"kubernetes.io/projected/4649d0af-fa58-47a7-8e50-c37a3236097c-kube-api-access-zmm9x\") pod \"payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:21.625998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.625968 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4649d0af-fa58-47a7-8e50-c37a3236097c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:21.727748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.727651 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4649d0af-fa58-47a7-8e50-c37a3236097c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:21.727917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.727758 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmm9x\" (UniqueName: \"kubernetes.io/projected/4649d0af-fa58-47a7-8e50-c37a3236097c-kube-api-access-zmm9x\") pod \"payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:21.728513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.728484 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4649d0af-fa58-47a7-8e50-c37a3236097c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:21.753535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.753505 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmm9x\" (UniqueName: \"kubernetes.io/projected/4649d0af-fa58-47a7-8e50-c37a3236097c-kube-api-access-zmm9x\") pod \"payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:21.772528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:21.772496 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:22.089685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:22.089655 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn"] Oct 31 21:34:22.091295 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:34:22.091265 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4649d0af_fa58_47a7_8e50_c37a3236097c.slice/crio-70a5f1ec3a6b61447d6cadf437d771e01a3f858d3b9819e8307573dec79c8a17 WatchSource:0}: Error finding container 70a5f1ec3a6b61447d6cadf437d771e01a3f858d3b9819e8307573dec79c8a17: Status 404 returned error can't find the container with id 70a5f1ec3a6b61447d6cadf437d771e01a3f858d3b9819e8307573dec79c8a17 Oct 31 21:34:22.266388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:22.266351 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" event={"ID":"4649d0af-fa58-47a7-8e50-c37a3236097c","Type":"ContainerStarted","Data":"f4f8d4abec8be42ddd5906c3d5be5ef810ae4a576e4d6bf1e2fa41614be4894f"} Oct 31 21:34:22.266575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:22.266425 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:22.266575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:22.266447 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" event={"ID":"4649d0af-fa58-47a7-8e50-c37a3236097c","Type":"ContainerStarted","Data":"70a5f1ec3a6b61447d6cadf437d771e01a3f858d3b9819e8307573dec79c8a17"} Oct 31 21:34:22.308745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:22.308690 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" podStartSLOduration=1.308677158 podStartE2EDuration="1.308677158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:34:22.307794787 +0000 UTC m=+1102.432902017" watchObservedRunningTime="2026-10-31 21:34:22.308677158 +0000 UTC m=+1102.433784439" Oct 31 21:34:31.310332 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:34:31.310281 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:34:33.275486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:33.275452 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:34:38.792324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.790875 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl"] Oct 31 21:34:38.810326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.807580 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:38.822333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.820983 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl"] Oct 31 21:34:38.831332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.830379 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-248b29-serving-cert\"" Oct 31 21:34:38.905336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.902622 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4t5c\" (UniqueName: \"kubernetes.io/projected/a6070ebd-cd3c-4d68-860b-3f7075efe67e-kube-api-access-n4t5c\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:38.905336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.902672 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:38.905336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.902698 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-metrics-certs\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:38.905336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:38.902834 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6070ebd-cd3c-4d68-860b-3f7075efe67e-openshift-service-ca\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.056397 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.025241 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6070ebd-cd3c-4d68-860b-3f7075efe67e-openshift-service-ca\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.079580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.073083 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t5c\" (UniqueName: \"kubernetes.io/projected/a6070ebd-cd3c-4d68-860b-3f7075efe67e-kube-api-access-n4t5c\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.079580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.073139 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.079580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.073173 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-metrics-certs\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.079580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.060157 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6070ebd-cd3c-4d68-860b-3f7075efe67e-openshift-service-ca\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.079580 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:34:39.074537 4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-serving-cert: secret "maas-api-e2e-api-b-248b29-serving-cert" not found Oct 31 21:34:39.079580 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:34:39.074593 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls podName:a6070ebd-cd3c-4d68-860b-3f7075efe67e nodeName:}" failed. No retries permitted until 2026-10-31 21:34:39.57457638 +0000 UTC m=+1119.699683604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls") pod "maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" (UID: "a6070ebd-cd3c-4d68-860b-3f7075efe67e") : secret "maas-api-e2e-api-b-248b29-serving-cert" not found Oct 31 21:34:39.123263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.123230 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4t5c\" (UniqueName: \"kubernetes.io/projected/a6070ebd-cd3c-4d68-860b-3f7075efe67e-kube-api-access-n4t5c\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.149326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.126872 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-metrics-certs\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.591415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.591378 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.618924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.618894 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls\") pod \"maas-api-e2e-api-b-248b29-c94f76d74-ll7tl\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:39.740326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:39.739542 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:40.462185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:40.462151 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl"] Oct 31 21:34:41.403643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:41.403599 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" event={"ID":"a6070ebd-cd3c-4d68-860b-3f7075efe67e","Type":"ContainerStarted","Data":"80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5"} Oct 31 21:34:41.403807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:41.403675 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" event={"ID":"a6070ebd-cd3c-4d68-860b-3f7075efe67e","Type":"ContainerStarted","Data":"1ef451bfe2ced9aeaf31d0b736d2f3260cb074808683a8d8e2bb181dc6a61ed8"} Oct 31 21:34:41.403807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:41.403711 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:41.446596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:41.446538 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" podStartSLOduration=3.072052741 podStartE2EDuration="3.446520806s" totalImagesPullingTime="374.468065ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:34:41.446040425 +0000 UTC m=+1121.571147655" watchObservedRunningTime="2026-10-31 21:34:41.446520806 +0000 UTC m=+1121.571628040" Oct 31 21:34:46.151979 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.151886 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc"] Oct 31 21:34:46.156551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.156528 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.174559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.174535 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-084d807f-openshift-default-dockercfg-8hmhd\"" Oct 31 21:34:46.190909 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.190879 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc"] Oct 31 21:34:46.313977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.313940 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.313984 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-credential-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.314021 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-certs\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.314037 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-istio-token\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.314119 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d59ef2e-382a-44e7-82b0-615d84578987-istiod-ca-cert\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.314168 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgnzl\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-kube-api-access-lgnzl\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.314254 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-data\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.314349 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-envoy\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.314562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.314488 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d59ef2e-382a-44e7-82b0-615d84578987-istio-podinfo\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.415711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.415609 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgnzl\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-kube-api-access-lgnzl\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.415711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.415698 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-data\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.415955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.415747 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-envoy\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.415955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.415826 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d59ef2e-382a-44e7-82b0-615d84578987-istio-podinfo\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.415992 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416016 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-credential-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416054 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-certs\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416079 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-istio-token\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416110 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d59ef2e-382a-44e7-82b0-615d84578987-istiod-ca-cert\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416164 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-data\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416295 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416532 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-credential-socket\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416573 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-certs\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.416837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.416816 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d59ef2e-382a-44e7-82b0-615d84578987-istiod-ca-cert\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.418162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.418135 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-envoy\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.418612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.418594 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d59ef2e-382a-44e7-82b0-615d84578987-istio-podinfo\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.439412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.439377 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-istio-token\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.439584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.439564 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgnzl\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-kube-api-access-lgnzl\") pod \"e2e-mt-084d807f-openshift-default-84c8676869-9pvlc\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.470510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.470472 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:46.804970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.804936 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc"] Oct 31 21:34:46.806904 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:34:46.806875 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d59ef2e_382a_44e7_82b0_615d84578987.slice/crio-afb5139f5e5f640ec0e96b6a3069daea47be664a9b1aebcfbe85682da79473bf WatchSource:0}: Error finding container afb5139f5e5f640ec0e96b6a3069daea47be664a9b1aebcfbe85682da79473bf: Status 404 returned error can't find the container with id afb5139f5e5f640ec0e96b6a3069daea47be664a9b1aebcfbe85682da79473bf Oct 31 21:34:46.809430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.809397 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:34:46.809499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.809464 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:34:46.809535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:46.809496 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:34:47.415471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:47.415442 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:34:47.443674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:47.443638 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" event={"ID":"5d59ef2e-382a-44e7-82b0-615d84578987","Type":"ContainerStarted","Data":"a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006"} Oct 31 21:34:47.443832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:47.443698 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" event={"ID":"5d59ef2e-382a-44e7-82b0-615d84578987","Type":"ContainerStarted","Data":"afb5139f5e5f640ec0e96b6a3069daea47be664a9b1aebcfbe85682da79473bf"} Oct 31 21:34:47.470808 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:47.470775 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:47.523495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:47.523454 4294 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:34:47.523690 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:47.523521 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 21:34:47.560544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:47.560485 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podStartSLOduration=1.560465802 podStartE2EDuration="1.560465802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:34:47.555240213 +0000 UTC m=+1127.680347444" watchObservedRunningTime="2026-10-31 21:34:47.560465802 +0000 UTC m=+1127.685573038" Oct 31 21:34:48.471274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:48.471239 4294 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:34:48.471811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:48.471324 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 21:34:49.471121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:49.471083 4294 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:34:49.471297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:49.471151 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 21:34:50.471047 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:50.471012 4294 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:34:50.471264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:50.471090 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 21:34:51.493326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:51.492739 4294 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:34:51.493326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:51.492805 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 21:34:52.478458 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:52.478419 4294 patch_prober.go:29] interesting pod/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 21:34:52.478772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:52.478737 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 21:34:53.475878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:53.475846 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:53.476389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:53.476142 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:34:53.476934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:34:53.476912 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:35:00.310201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.310164 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796"] Oct 31 21:35:00.315553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.315527 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.335070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.335044 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796"] Oct 31 21:35:00.343069 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.343045 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-08ddf0\"" Oct 31 21:35:00.377795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.377763 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswwg\" (UniqueName: \"kubernetes.io/projected/19617a06-32d8-4546-9bec-c44088963e6c-kube-api-access-zswwg\") pod \"payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.377795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.377799 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19617a06-32d8-4546-9bec-c44088963e6c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.409675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.409636 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-08ddf0-dockercfg-hvksz\"" Oct 31 21:35:00.486197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.480397 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswwg\" (UniqueName: \"kubernetes.io/projected/19617a06-32d8-4546-9bec-c44088963e6c-kube-api-access-zswwg\") pod \"payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.486197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.480445 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19617a06-32d8-4546-9bec-c44088963e6c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.486197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.481423 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19617a06-32d8-4546-9bec-c44088963e6c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.513893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.513859 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswwg\" (UniqueName: \"kubernetes.io/projected/19617a06-32d8-4546-9bec-c44088963e6c-kube-api-access-zswwg\") pod \"payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.633728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.633638 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:00.952579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:00.952550 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796"] Oct 31 21:35:00.953722 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:35:00.953696 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19617a06_32d8_4546_9bec_c44088963e6c.slice/crio-32e2daccb5992e5a050e406b4b601c6b7a26485a67018ecacb3035e3afb9d758 WatchSource:0}: Error finding container 32e2daccb5992e5a050e406b4b601c6b7a26485a67018ecacb3035e3afb9d758: Status 404 returned error can't find the container with id 32e2daccb5992e5a050e406b4b601c6b7a26485a67018ecacb3035e3afb9d758 Oct 31 21:35:01.320349 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:35:01.318699 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:35:01.556986 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:01.556941 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" event={"ID":"19617a06-32d8-4546-9bec-c44088963e6c","Type":"ContainerStarted","Data":"856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36"} Oct 31 21:35:01.557161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:01.556996 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" event={"ID":"19617a06-32d8-4546-9bec-c44088963e6c","Type":"ContainerStarted","Data":"32e2daccb5992e5a050e406b4b601c6b7a26485a67018ecacb3035e3afb9d758"} Oct 31 21:35:01.557161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:01.557024 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:01.599807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:01.599706 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" podStartSLOduration=1.59969288 podStartE2EDuration="1.59969288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:35:01.598016426 +0000 UTC m=+1141.723123658" watchObservedRunningTime="2026-10-31 21:35:01.59969288 +0000 UTC m=+1141.724800110" Oct 31 21:35:12.567623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:12.567590 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:35:13.309850 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.309820 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7549-f0c17f"] Oct 31 21:35:13.313923 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.313908 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:13.330427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.330399 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-f0c17f"] Oct 31 21:35:13.338382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.338360 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dsnkq\"" Oct 31 21:35:13.362490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.362463 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpn8l\" (UniqueName: \"kubernetes.io/projected/b25d964b-ff84-4252-a01d-17d2631ff18b-kube-api-access-mpn8l\") pod \"test-curl-7549-f0c17f\" (UID: \"b25d964b-ff84-4252-a01d-17d2631ff18b\") " pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:13.463977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.463948 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpn8l\" (UniqueName: \"kubernetes.io/projected/b25d964b-ff84-4252-a01d-17d2631ff18b-kube-api-access-mpn8l\") pod \"test-curl-7549-f0c17f\" (UID: \"b25d964b-ff84-4252-a01d-17d2631ff18b\") " pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:13.487878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.487854 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpn8l\" (UniqueName: \"kubernetes.io/projected/b25d964b-ff84-4252-a01d-17d2631ff18b-kube-api-access-mpn8l\") pod \"test-curl-7549-f0c17f\" (UID: \"b25d964b-ff84-4252-a01d-17d2631ff18b\") " pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:13.626113 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.626011 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:13.937752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:13.937728 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-f0c17f"] Oct 31 21:35:13.939339 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:35:13.939295 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb25d964b_ff84_4252_a01d_17d2631ff18b.slice/crio-96051d0043434e560ea66cfbf3d3e49d81e0cf6a9366315c20f2e7cd23536c01 WatchSource:0}: Error finding container 96051d0043434e560ea66cfbf3d3e49d81e0cf6a9366315c20f2e7cd23536c01: Status 404 returned error can't find the container with id 96051d0043434e560ea66cfbf3d3e49d81e0cf6a9366315c20f2e7cd23536c01 Oct 31 21:35:14.640258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:14.640224 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-f0c17f" event={"ID":"b25d964b-ff84-4252-a01d-17d2631ff18b","Type":"ContainerStarted","Data":"96051d0043434e560ea66cfbf3d3e49d81e0cf6a9366315c20f2e7cd23536c01"} Oct 31 21:35:17.763994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:17.763955 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-f0c17f" event={"ID":"b25d964b-ff84-4252-a01d-17d2631ff18b","Type":"ContainerStarted","Data":"e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36"} Oct 31 21:35:18.785637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:18.785572 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7549-f0c17f" podStartSLOduration=2.390995239 podStartE2EDuration="5.785553394s" totalImagesPullingTime="3.394558155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:35:17.821510735 +0000 UTC m=+1157.946617970" watchObservedRunningTime="2026-10-31 21:35:18.785553394 +0000 UTC m=+1158.910660624" Oct 31 21:35:18.793418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:18.793384 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7549-f0c17f"] Oct 31 21:35:18.793570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:18.793472 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7549-f0c17f"] Oct 31 21:35:19.779187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:19.779122 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7549-f0c17f" podUID="b25d964b-ff84-4252-a01d-17d2631ff18b" containerName="test-curl-7549-f0c17f" containerID="cri-o://e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36" gracePeriod=2 Oct 31 21:35:19.792595 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:35:19.792553 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7549-f0c17f\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b25d964b-ff84-4252-a01d-17d2631ff18b" pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:19.994152 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:35:19.994113 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7549-f0c17f\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b25d964b-ff84-4252-a01d-17d2631ff18b" pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:21.977521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:21.977496 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:22.025491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.025462 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25d964b-ff84-4252-a01d-17d2631ff18b-kube-api-access-mpn8l\" (UniqueName: \"kubernetes.io/projected/b25d964b-ff84-4252-a01d-17d2631ff18b-kube-api-access-mpn8l\") pod \"b25d964b-ff84-4252-a01d-17d2631ff18b\" (UID: \"b25d964b-ff84-4252-a01d-17d2631ff18b\") " Oct 31 21:35:22.027548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.027525 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25d964b-ff84-4252-a01d-17d2631ff18b-kube-api-access-mpn8l" pod "b25d964b-ff84-4252-a01d-17d2631ff18b" (UID: "b25d964b-ff84-4252-a01d-17d2631ff18b"). InnerVolumeSpecName "kube-api-access-mpn8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:22.127104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.127057 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpn8l\" (UniqueName: \"kubernetes.io/projected/b25d964b-ff84-4252-a01d-17d2631ff18b-kube-api-access-mpn8l\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:35:22.797077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.797044 4294 generic.go:356] "Generic (PLEG): container finished" podID="b25d964b-ff84-4252-a01d-17d2631ff18b" containerID="e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36" exitCode=137 Oct 31 21:35:22.797077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.797083 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-f0c17f" Oct 31 21:35:22.797439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.797145 4294 scope.go:122] "RemoveContainer" containerID="e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36" Oct 31 21:35:22.807564 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.807544 4294 scope.go:122] "RemoveContainer" containerID="e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36" Oct 31 21:35:22.807794 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:35:22.807776 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36\": container with ID starting with e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36 not found: ID does not exist" containerID="e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36" Oct 31 21:35:22.807842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:22.807802 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36"} err="failed to get container status \"e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36\": rpc error: code = NotFound desc = could not find container \"e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36\": container with ID starting with e24ac1cc81f48441ee06fecbed4bbf0fc9d5930fd3f0b51966623364fe4fce36 not found: ID does not exist" Oct 31 21:35:23.987515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:23.987484 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25d964b-ff84-4252-a01d-17d2631ff18b" path="/var/lib/kubelet/pods/b25d964b-ff84-4252-a01d-17d2631ff18b/volumes" Oct 31 21:35:31.311347 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:35:31.311285 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:35:45.190286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:45.190251 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb"] Oct 31 21:35:45.857064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:45.857023 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx"] Oct 31 21:35:45.857349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:45.857318 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" podUID="e592a26d-cc33-4400-9b83-d5a2f204a632" containerName="maas-api" containerID="cri-o://246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc" gracePeriod=30 Oct 31 21:35:46.280352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.279480 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:35:46.366864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.366818 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-metrics-certs\") pod \"e592a26d-cc33-4400-9b83-d5a2f204a632\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " Oct 31 21:35:46.367057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.366876 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-maas-api-tls\") pod \"e592a26d-cc33-4400-9b83-d5a2f204a632\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " Oct 31 21:35:46.367144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.367111 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e592a26d-cc33-4400-9b83-d5a2f204a632-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e592a26d-cc33-4400-9b83-d5a2f204a632-openshift-service-ca\") pod \"e592a26d-cc33-4400-9b83-d5a2f204a632\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " Oct 31 21:35:46.367246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.367226 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e592a26d-cc33-4400-9b83-d5a2f204a632-kube-api-access-xbnpz\" (UniqueName: \"kubernetes.io/projected/e592a26d-cc33-4400-9b83-d5a2f204a632-kube-api-access-xbnpz\") pod \"e592a26d-cc33-4400-9b83-d5a2f204a632\" (UID: \"e592a26d-cc33-4400-9b83-d5a2f204a632\") " Oct 31 21:35:46.367601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.367573 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e592a26d-cc33-4400-9b83-d5a2f204a632-openshift-service-ca" pod "e592a26d-cc33-4400-9b83-d5a2f204a632" (UID: "e592a26d-cc33-4400-9b83-d5a2f204a632"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:46.368132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.368107 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e592a26d-cc33-4400-9b83-d5a2f204a632-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:35:46.370008 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.369966 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-maas-api-tls" pod "e592a26d-cc33-4400-9b83-d5a2f204a632" (UID: "e592a26d-cc33-4400-9b83-d5a2f204a632"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:35:46.370277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.370242 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-metrics-certs" pod "e592a26d-cc33-4400-9b83-d5a2f204a632" (UID: "e592a26d-cc33-4400-9b83-d5a2f204a632"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:35:46.372263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.372239 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e592a26d-cc33-4400-9b83-d5a2f204a632-kube-api-access-xbnpz" pod "e592a26d-cc33-4400-9b83-d5a2f204a632" (UID: "e592a26d-cc33-4400-9b83-d5a2f204a632"). InnerVolumeSpecName "kube-api-access-xbnpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:46.473924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.469527 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:35:46.473924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.469563 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e592a26d-cc33-4400-9b83-d5a2f204a632-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:35:46.473924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.469577 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbnpz\" (UniqueName: \"kubernetes.io/projected/e592a26d-cc33-4400-9b83-d5a2f204a632-kube-api-access-xbnpz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:35:46.982034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.982004 4294 generic.go:356] "Generic (PLEG): container finished" podID="e592a26d-cc33-4400-9b83-d5a2f204a632" containerID="246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc" exitCode=0 Oct 31 21:35:46.982226 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.982087 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" Oct 31 21:35:46.982226 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.982096 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" event={"ID":"e592a26d-cc33-4400-9b83-d5a2f204a632","Type":"ContainerDied","Data":"246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc"} Oct 31 21:35:46.982226 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.982209 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx" event={"ID":"e592a26d-cc33-4400-9b83-d5a2f204a632","Type":"ContainerDied","Data":"b65bfaad09dd56d1688049915eb82cddbcfd91e66d00184c144afc1b3ee32a13"} Oct 31 21:35:46.982375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.982232 4294 scope.go:122] "RemoveContainer" containerID="246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc" Oct 31 21:35:46.993168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.993143 4294 scope.go:122] "RemoveContainer" containerID="246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc" Oct 31 21:35:46.993475 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:35:46.993454 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc\": container with ID starting with 246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc not found: ID does not exist" containerID="246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc" Oct 31 21:35:46.993556 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:46.993484 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc"} err="failed to get container status \"246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc\": rpc error: code = NotFound desc = could not find container \"246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc\": container with ID starting with 246e042451c293c454e884bd1fd610c319583633bb3ea77cfb5ddf6338b1e6dc not found: ID does not exist" Oct 31 21:35:47.033956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.033930 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx"] Oct 31 21:35:47.045973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.045943 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7ac3c8-f548f74c7-v9lsx"] Oct 31 21:35:47.197745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.197705 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7"] Oct 31 21:35:47.198839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.198785 4294 cpu_manager.go:414] "containerMap: removing container" podUID="b25d964b-ff84-4252-a01d-17d2631ff18b" containerName="test-curl-7549-f0c17f" Oct 31 21:35:47.198839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.198812 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25d964b-ff84-4252-a01d-17d2631ff18b" containerName="test-curl-7549-f0c17f" Oct 31 21:35:47.199121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.198853 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e592a26d-cc33-4400-9b83-d5a2f204a632" containerName="maas-api" Oct 31 21:35:47.199121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.198863 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e592a26d-cc33-4400-9b83-d5a2f204a632" containerName="maas-api" Oct 31 21:35:47.199121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.199027 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25d964b-ff84-4252-a01d-17d2631ff18b" containerName="test-curl-7549-f0c17f" Oct 31 21:35:47.199121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.199043 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e592a26d-cc33-4400-9b83-d5a2f204a632" containerName="maas-api" Oct 31 21:35:47.267875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.267843 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.290854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.290144 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-43fb1b\"" Oct 31 21:35:47.290854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.290814 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-43fb1b-dockercfg-nf24q\"" Oct 31 21:35:47.358369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.346940 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7"] Oct 31 21:35:47.388628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.388590 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fc09a6-3733-4d20-a197-30304074e9ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.389115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.389085 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5zrr\" (UniqueName: \"kubernetes.io/projected/91fc09a6-3733-4d20-a197-30304074e9ce-kube-api-access-l5zrr\") pod \"payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.491018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.490968 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fc09a6-3733-4d20-a197-30304074e9ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.491229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.491204 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5zrr\" (UniqueName: \"kubernetes.io/projected/91fc09a6-3733-4d20-a197-30304074e9ce-kube-api-access-l5zrr\") pod \"payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.491779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.491757 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fc09a6-3733-4d20-a197-30304074e9ce-plugins-config-volume\") pod \"payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.517416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.517384 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5zrr\" (UniqueName: \"kubernetes.io/projected/91fc09a6-3733-4d20-a197-30304074e9ce-kube-api-access-l5zrr\") pod \"payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.596763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.596676 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.912426 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.912398 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7"] Oct 31 21:35:47.914546 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:35:47.914516 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fc09a6_3733_4d20_a197_30304074e9ce.slice/crio-e59fa3f4303241715d22bbac71823540a1f4bd7040f09d8e4fd8440c4d4630b0 WatchSource:0}: Error finding container e59fa3f4303241715d22bbac71823540a1f4bd7040f09d8e4fd8440c4d4630b0: Status 404 returned error can't find the container with id e59fa3f4303241715d22bbac71823540a1f4bd7040f09d8e4fd8440c4d4630b0 Oct 31 21:35:47.986286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.986262 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e592a26d-cc33-4400-9b83-d5a2f204a632" path="/var/lib/kubelet/pods/e592a26d-cc33-4400-9b83-d5a2f204a632/volumes" Oct 31 21:35:47.990049 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.990014 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" event={"ID":"91fc09a6-3733-4d20-a197-30304074e9ce","Type":"ContainerStarted","Data":"81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b"} Oct 31 21:35:47.990161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.990083 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:35:47.990161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:47.990111 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" event={"ID":"91fc09a6-3733-4d20-a197-30304074e9ce","Type":"ContainerStarted","Data":"e59fa3f4303241715d22bbac71823540a1f4bd7040f09d8e4fd8440c4d4630b0"} Oct 31 21:35:48.032426 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:48.032373 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" podStartSLOduration=1.032358455 podStartE2EDuration="1.032358455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:35:48.029831448 +0000 UTC m=+1188.154938680" watchObservedRunningTime="2026-10-31 21:35:48.032358455 +0000 UTC m=+1188.157465686" Oct 31 21:35:50.191064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.191016 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" podUID="95989257-2156-4ffa-b83a-73ff9c0701a4" containerName="payload-pre-processing" containerID="cri-o://64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86" gracePeriod=25 Oct 31 21:35:50.469380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.469295 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:35:50.543849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.543817 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95989257-2156-4ffa-b83a-73ff9c0701a4-kube-api-access-vqhf8\" (UniqueName: \"kubernetes.io/projected/95989257-2156-4ffa-b83a-73ff9c0701a4-kube-api-access-vqhf8\") pod \"95989257-2156-4ffa-b83a-73ff9c0701a4\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " Oct 31 21:35:50.544043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.544023 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95989257-2156-4ffa-b83a-73ff9c0701a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95989257-2156-4ffa-b83a-73ff9c0701a4-plugins-config-volume\") pod \"95989257-2156-4ffa-b83a-73ff9c0701a4\" (UID: \"95989257-2156-4ffa-b83a-73ff9c0701a4\") " Oct 31 21:35:50.544411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.544388 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95989257-2156-4ffa-b83a-73ff9c0701a4-plugins-config-volume" pod "95989257-2156-4ffa-b83a-73ff9c0701a4" (UID: "95989257-2156-4ffa-b83a-73ff9c0701a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:35:50.544630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.544614 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95989257-2156-4ffa-b83a-73ff9c0701a4-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:35:50.546212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.546180 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95989257-2156-4ffa-b83a-73ff9c0701a4-kube-api-access-vqhf8" pod "95989257-2156-4ffa-b83a-73ff9c0701a4" (UID: "95989257-2156-4ffa-b83a-73ff9c0701a4"). InnerVolumeSpecName "kube-api-access-vqhf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:35:50.645995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:50.645970 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqhf8\" (UniqueName: \"kubernetes.io/projected/95989257-2156-4ffa-b83a-73ff9c0701a4-kube-api-access-vqhf8\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:35:51.010912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.010876 4294 generic.go:356] "Generic (PLEG): container finished" podID="95989257-2156-4ffa-b83a-73ff9c0701a4" containerID="64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86" exitCode=0 Oct 31 21:35:51.011080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.010920 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" Oct 31 21:35:51.011080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.010959 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" event={"ID":"95989257-2156-4ffa-b83a-73ff9c0701a4","Type":"ContainerDied","Data":"64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86"} Oct 31 21:35:51.011080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.011021 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb" event={"ID":"95989257-2156-4ffa-b83a-73ff9c0701a4","Type":"ContainerDied","Data":"abd5add6ae1517c207114d589a4f2a705e92a153b23e2a911994beaac01d1027"} Oct 31 21:35:51.011080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.011057 4294 scope.go:122] "RemoveContainer" containerID="64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86" Oct 31 21:35:51.028222 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.028143 4294 scope.go:122] "RemoveContainer" containerID="64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86" Oct 31 21:35:51.028555 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:35:51.028535 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86\": container with ID starting with 64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86 not found: ID does not exist" containerID="64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86" Oct 31 21:35:51.028629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.028566 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86"} err="failed to get container status \"64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86\": rpc error: code = NotFound desc = could not find container \"64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86\": container with ID starting with 64f7620744e263b4dd654609b8270d8dfbd99e76b71d92caa606041183c25b86 not found: ID does not exist" Oct 31 21:35:51.067241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.067215 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb"] Oct 31 21:35:51.070030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.070010 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-371531-56c4759659-xfppb"] Oct 31 21:35:51.985566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:51.985527 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95989257-2156-4ffa-b83a-73ff9c0701a4" path="/var/lib/kubelet/pods/95989257-2156-4ffa-b83a-73ff9c0701a4/volumes" Oct 31 21:35:58.999786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:35:58.999753 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:36:00.086784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:00.086752 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:36:00.091111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:00.091075 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:36:01.315139 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:01.315107 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:36:03.119092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.119058 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt"] Oct 31 21:36:03.119610 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.119293 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" podUID="d033fb75-e793-442f-a7d8-fc9bcccb1495" containerName="maas-api" containerID="cri-o://78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29" gracePeriod=30 Oct 31 21:36:03.532246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.531944 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:36:03.536167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.535736 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94"] Oct 31 21:36:03.536167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.535905 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" podUID="5f1567ac-8c97-4c54-a0b2-fdd6a321f435" containerName="maas-api" containerID="cri-o://3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f" gracePeriod=30 Oct 31 21:36:03.537775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.537748 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d033fb75-e793-442f-a7d8-fc9bcccb1495-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d033fb75-e793-442f-a7d8-fc9bcccb1495-openshift-service-ca\") pod \"d033fb75-e793-442f-a7d8-fc9bcccb1495\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " Oct 31 21:36:03.538186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.538127 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d033fb75-e793-442f-a7d8-fc9bcccb1495-kube-api-access-s8dj2\" (UniqueName: \"kubernetes.io/projected/d033fb75-e793-442f-a7d8-fc9bcccb1495-kube-api-access-s8dj2\") pod \"d033fb75-e793-442f-a7d8-fc9bcccb1495\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " Oct 31 21:36:03.539167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.539093 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls\") pod \"d033fb75-e793-442f-a7d8-fc9bcccb1495\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " Oct 31 21:36:03.539556 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.539520 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d033fb75-e793-442f-a7d8-fc9bcccb1495-openshift-service-ca" pod "d033fb75-e793-442f-a7d8-fc9bcccb1495" (UID: "d033fb75-e793-442f-a7d8-fc9bcccb1495"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:03.539820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.539776 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-metrics-certs\") pod \"d033fb75-e793-442f-a7d8-fc9bcccb1495\" (UID: \"d033fb75-e793-442f-a7d8-fc9bcccb1495\") " Oct 31 21:36:03.541588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.541557 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d033fb75-e793-442f-a7d8-fc9bcccb1495-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:03.542978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.542941 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d033fb75-e793-442f-a7d8-fc9bcccb1495-kube-api-access-s8dj2" pod "d033fb75-e793-442f-a7d8-fc9bcccb1495" (UID: "d033fb75-e793-442f-a7d8-fc9bcccb1495"). InnerVolumeSpecName "kube-api-access-s8dj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:03.549045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.549021 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-metrics-certs" pod "d033fb75-e793-442f-a7d8-fc9bcccb1495" (UID: "d033fb75-e793-442f-a7d8-fc9bcccb1495"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:03.550924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.550837 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls" pod "d033fb75-e793-442f-a7d8-fc9bcccb1495" (UID: "d033fb75-e793-442f-a7d8-fc9bcccb1495"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:03.681362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.676124 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8dj2\" (UniqueName: \"kubernetes.io/projected/d033fb75-e793-442f-a7d8-fc9bcccb1495-kube-api-access-s8dj2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:03.681362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.676168 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:03.681362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:03.676183 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d033fb75-e793-442f-a7d8-fc9bcccb1495-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:04.028885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.028859 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:36:04.116393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.113294 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-kube-api-access-bt97z\" (UniqueName: \"kubernetes.io/projected/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-kube-api-access-bt97z\") pod \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " Oct 31 21:36:04.116393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.113435 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls\") pod \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " Oct 31 21:36:04.116393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.113580 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-metrics-certs\") pod \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " Oct 31 21:36:04.116393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.113690 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-openshift-service-ca\") pod \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\" (UID: \"5f1567ac-8c97-4c54-a0b2-fdd6a321f435\") " Oct 31 21:36:04.116393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.114649 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-openshift-service-ca" pod "5f1567ac-8c97-4c54-a0b2-fdd6a321f435" (UID: "5f1567ac-8c97-4c54-a0b2-fdd6a321f435"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:04.119365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.117385 4294 generic.go:356] "Generic (PLEG): container finished" podID="5f1567ac-8c97-4c54-a0b2-fdd6a321f435" containerID="3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f" exitCode=0 Oct 31 21:36:04.119365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.117558 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" event={"ID":"5f1567ac-8c97-4c54-a0b2-fdd6a321f435","Type":"ContainerDied","Data":"3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f"} Oct 31 21:36:04.119365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.117608 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" event={"ID":"5f1567ac-8c97-4c54-a0b2-fdd6a321f435","Type":"ContainerDied","Data":"b0a3fd54aa300f1a3e45d961182e8717e6a68393d9066b77da9c3e309048600c"} Oct 31 21:36:04.119365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.117639 4294 scope.go:122] "RemoveContainer" containerID="3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f" Oct 31 21:36:04.119365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.117659 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94" Oct 31 21:36:04.119365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.117653 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-kube-api-access-bt97z" pod "5f1567ac-8c97-4c54-a0b2-fdd6a321f435" (UID: "5f1567ac-8c97-4c54-a0b2-fdd6a321f435"). InnerVolumeSpecName "kube-api-access-bt97z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:04.124752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.122978 4294 generic.go:356] "Generic (PLEG): container finished" podID="d033fb75-e793-442f-a7d8-fc9bcccb1495" containerID="78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29" exitCode=0 Oct 31 21:36:04.124752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.123082 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" event={"ID":"d033fb75-e793-442f-a7d8-fc9bcccb1495","Type":"ContainerDied","Data":"78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29"} Oct 31 21:36:04.124752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.123134 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" event={"ID":"d033fb75-e793-442f-a7d8-fc9bcccb1495","Type":"ContainerDied","Data":"a290850a378a782d87c9b6eea89cc34ba4aad2c213bdd65d4a646c7f0ffd5cd4"} Oct 31 21:36:04.124752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.123201 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt" Oct 31 21:36:04.156355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.154861 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls" pod "5f1567ac-8c97-4c54-a0b2-fdd6a321f435" (UID: "5f1567ac-8c97-4c54-a0b2-fdd6a321f435"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:04.156355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.154891 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-metrics-certs" pod "5f1567ac-8c97-4c54-a0b2-fdd6a321f435" (UID: "5f1567ac-8c97-4c54-a0b2-fdd6a321f435"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:04.188161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.188036 4294 scope.go:122] "RemoveContainer" containerID="3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f" Oct 31 21:36:04.189288 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:04.189017 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f\": container with ID starting with 3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f not found: ID does not exist" containerID="3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f" Oct 31 21:36:04.189288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.189047 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f"} err="failed to get container status \"3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f\": rpc error: code = NotFound desc = could not find container \"3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f\": container with ID starting with 3cbeee5aea48f3d0334a5076f8b74ca19f828be8c5cf68aabe00fa65455bf06f not found: ID does not exist" Oct 31 21:36:04.189288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.189075 4294 scope.go:122] "RemoveContainer" containerID="78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29" Oct 31 21:36:04.207163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.207144 4294 scope.go:122] "RemoveContainer" containerID="78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29" Oct 31 21:36:04.207498 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:04.207474 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29\": container with ID starting with 78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29 not found: ID does not exist" containerID="78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29" Oct 31 21:36:04.207611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.207507 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29"} err="failed to get container status \"78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29\": rpc error: code = NotFound desc = could not find container \"78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29\": container with ID starting with 78b8d2d1b250a75b8cae81b2b6f4af4cca687c2f82fb54c4f68f302035de9e29 not found: ID does not exist" Oct 31 21:36:04.215515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.215475 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt97z\" (UniqueName: \"kubernetes.io/projected/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-kube-api-access-bt97z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:04.215515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.215502 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:04.215515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.215519 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:04.215761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.215530 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f1567ac-8c97-4c54-a0b2-fdd6a321f435-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:04.238704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.238639 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt"] Oct 31 21:36:04.244849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.244803 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-49rqt"] Oct 31 21:36:04.466667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.466626 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94"] Oct 31 21:36:04.471846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:04.471818 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-371531-59df8fbf9f-hrq94"] Oct 31 21:36:05.987114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:05.987076 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1567ac-8c97-4c54-a0b2-fdd6a321f435" path="/var/lib/kubelet/pods/5f1567ac-8c97-4c54-a0b2-fdd6a321f435/volumes" Oct 31 21:36:05.987591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:05.987576 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d033fb75-e793-442f-a7d8-fc9bcccb1495" path="/var/lib/kubelet/pods/d033fb75-e793-442f-a7d8-fc9bcccb1495/volumes" Oct 31 21:36:06.019768 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:06.019741 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn"] Oct 31 21:36:11.020409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.020369 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" podUID="4649d0af-fa58-47a7-8e50-c37a3236097c" containerName="payload-pre-processing" containerID="cri-o://f4f8d4abec8be42ddd5906c3d5be5ef810ae4a576e4d6bf1e2fa41614be4894f" gracePeriod=25 Oct 31 21:36:11.172491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.172459 4294 generic.go:356] "Generic (PLEG): container finished" podID="4649d0af-fa58-47a7-8e50-c37a3236097c" containerID="f4f8d4abec8be42ddd5906c3d5be5ef810ae4a576e4d6bf1e2fa41614be4894f" exitCode=0 Oct 31 21:36:11.172648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.172531 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" event={"ID":"4649d0af-fa58-47a7-8e50-c37a3236097c","Type":"ContainerDied","Data":"f4f8d4abec8be42ddd5906c3d5be5ef810ae4a576e4d6bf1e2fa41614be4894f"} Oct 31 21:36:11.290362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.290338 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:36:11.414386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.414343 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4649d0af-fa58-47a7-8e50-c37a3236097c-kube-api-access-zmm9x\" (UniqueName: \"kubernetes.io/projected/4649d0af-fa58-47a7-8e50-c37a3236097c-kube-api-access-zmm9x\") pod \"4649d0af-fa58-47a7-8e50-c37a3236097c\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " Oct 31 21:36:11.414562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.414437 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4649d0af-fa58-47a7-8e50-c37a3236097c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4649d0af-fa58-47a7-8e50-c37a3236097c-plugins-config-volume\") pod \"4649d0af-fa58-47a7-8e50-c37a3236097c\" (UID: \"4649d0af-fa58-47a7-8e50-c37a3236097c\") " Oct 31 21:36:11.414798 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.414773 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4649d0af-fa58-47a7-8e50-c37a3236097c-plugins-config-volume" pod "4649d0af-fa58-47a7-8e50-c37a3236097c" (UID: "4649d0af-fa58-47a7-8e50-c37a3236097c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:11.415028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.415011 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4649d0af-fa58-47a7-8e50-c37a3236097c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:11.416584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.416552 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4649d0af-fa58-47a7-8e50-c37a3236097c-kube-api-access-zmm9x" pod "4649d0af-fa58-47a7-8e50-c37a3236097c" (UID: "4649d0af-fa58-47a7-8e50-c37a3236097c"). InnerVolumeSpecName "kube-api-access-zmm9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:11.538744 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:11.525108 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmm9x\" (UniqueName: \"kubernetes.io/projected/4649d0af-fa58-47a7-8e50-c37a3236097c-kube-api-access-zmm9x\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:12.183909 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:12.183869 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" event={"ID":"4649d0af-fa58-47a7-8e50-c37a3236097c","Type":"ContainerDied","Data":"70a5f1ec3a6b61447d6cadf437d771e01a3f858d3b9819e8307573dec79c8a17"} Oct 31 21:36:12.184374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:12.183947 4294 scope.go:122] "RemoveContainer" containerID="f4f8d4abec8be42ddd5906c3d5be5ef810ae4a576e4d6bf1e2fa41614be4894f" Oct 31 21:36:12.184374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:12.184119 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn" Oct 31 21:36:12.240213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:12.240179 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn"] Oct 31 21:36:12.252206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:12.252177 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-248b29-56d94578bc-xhmmn"] Oct 31 21:36:13.986471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:13.986399 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4649d0af-fa58-47a7-8e50-c37a3236097c" path="/var/lib/kubelet/pods/4649d0af-fa58-47a7-8e50-c37a3236097c/volumes" Oct 31 21:36:16.585363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585329 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r"] Oct 31 21:36:16.585885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585871 4294 cpu_manager.go:414] "containerMap: removing container" podUID="95989257-2156-4ffa-b83a-73ff9c0701a4" containerName="payload-pre-processing" Oct 31 21:36:16.585933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585886 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95989257-2156-4ffa-b83a-73ff9c0701a4" containerName="payload-pre-processing" Oct 31 21:36:16.585933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585910 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5f1567ac-8c97-4c54-a0b2-fdd6a321f435" containerName="maas-api" Oct 31 21:36:16.585933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585915 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1567ac-8c97-4c54-a0b2-fdd6a321f435" containerName="maas-api" Oct 31 21:36:16.585933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585930 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d033fb75-e793-442f-a7d8-fc9bcccb1495" containerName="maas-api" Oct 31 21:36:16.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585935 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d033fb75-e793-442f-a7d8-fc9bcccb1495" containerName="maas-api" Oct 31 21:36:16.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585942 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4649d0af-fa58-47a7-8e50-c37a3236097c" containerName="payload-pre-processing" Oct 31 21:36:16.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.585947 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4649d0af-fa58-47a7-8e50-c37a3236097c" containerName="payload-pre-processing" Oct 31 21:36:16.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.586026 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="95989257-2156-4ffa-b83a-73ff9c0701a4" containerName="payload-pre-processing" Oct 31 21:36:16.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.586039 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1567ac-8c97-4c54-a0b2-fdd6a321f435" containerName="maas-api" Oct 31 21:36:16.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.586046 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d033fb75-e793-442f-a7d8-fc9bcccb1495" containerName="maas-api" Oct 31 21:36:16.586057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.586053 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4649d0af-fa58-47a7-8e50-c37a3236097c" containerName="payload-pre-processing" Oct 31 21:36:16.590720 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.590698 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.608375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.608345 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r"] Oct 31 21:36:16.612875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.612833 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbc8b\" (UniqueName: \"kubernetes.io/projected/f265ffe5-40ed-409b-8358-a0ec7732c55f-kube-api-access-mbc8b\") pod \"payload-processing-e2e-mt-084d807f-7648554b69-4gw5r\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.613009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.612907 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f265ffe5-40ed-409b-8358-a0ec7732c55f-plugins-config-volume\") pod \"payload-processing-e2e-mt-084d807f-7648554b69-4gw5r\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.617337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.616475 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-084d807f\"" Oct 31 21:36:16.623849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.623823 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx"] Oct 31 21:36:16.628414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.628395 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:16.657031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.656984 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx"] Oct 31 21:36:16.706995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.706955 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-084d807f-dockercfg-kknkx\"" Oct 31 21:36:16.714170 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.714131 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad33938e-2fbb-47f7-ae05-1231d69634a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:16.714453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.714430 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbc8b\" (UniqueName: \"kubernetes.io/projected/f265ffe5-40ed-409b-8358-a0ec7732c55f-kube-api-access-mbc8b\") pod \"payload-processing-e2e-mt-084d807f-7648554b69-4gw5r\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.714532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.714504 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f265ffe5-40ed-409b-8358-a0ec7732c55f-plugins-config-volume\") pod \"payload-processing-e2e-mt-084d807f-7648554b69-4gw5r\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.714627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.714608 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6kpk\" (UniqueName: \"kubernetes.io/projected/ad33938e-2fbb-47f7-ae05-1231d69634a6-kube-api-access-c6kpk\") pod \"payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:16.715337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.715316 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f265ffe5-40ed-409b-8358-a0ec7732c55f-plugins-config-volume\") pod \"payload-processing-e2e-mt-084d807f-7648554b69-4gw5r\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.738519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.738490 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbc8b\" (UniqueName: \"kubernetes.io/projected/f265ffe5-40ed-409b-8358-a0ec7732c55f-kube-api-access-mbc8b\") pod \"payload-processing-e2e-mt-084d807f-7648554b69-4gw5r\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.840382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.819677 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6kpk\" (UniqueName: \"kubernetes.io/projected/ad33938e-2fbb-47f7-ae05-1231d69634a6-kube-api-access-c6kpk\") pod \"payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:16.840382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.819787 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad33938e-2fbb-47f7-ae05-1231d69634a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:16.840382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.820782 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad33938e-2fbb-47f7-ae05-1231d69634a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:16.852909 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.852843 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6kpk\" (UniqueName: \"kubernetes.io/projected/ad33938e-2fbb-47f7-ae05-1231d69634a6-kube-api-access-c6kpk\") pod \"payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:16.907328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.905708 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:16.946706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:16.946162 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:17.283021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:17.282976 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r"] Oct 31 21:36:17.294576 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:36:17.294522 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf265ffe5_40ed_409b_8358_a0ec7732c55f.slice/crio-e3f74b1c98dc537283faa937b25392a92eb109f7c6f53fad5f2dcc7ff4f76cd0 WatchSource:0}: Error finding container e3f74b1c98dc537283faa937b25392a92eb109f7c6f53fad5f2dcc7ff4f76cd0: Status 404 returned error can't find the container with id e3f74b1c98dc537283faa937b25392a92eb109f7c6f53fad5f2dcc7ff4f76cd0 Oct 31 21:36:17.341684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:17.340455 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx"] Oct 31 21:36:17.345841 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:36:17.345775 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad33938e_2fbb_47f7_ae05_1231d69634a6.slice/crio-b752f11db3748b9abf4893f30831004c469571d823c64917047d8782b2b2bbac WatchSource:0}: Error finding container b752f11db3748b9abf4893f30831004c469571d823c64917047d8782b2b2bbac: Status 404 returned error can't find the container with id b752f11db3748b9abf4893f30831004c469571d823c64917047d8782b2b2bbac Oct 31 21:36:18.261537 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.261479 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" event={"ID":"ad33938e-2fbb-47f7-ae05-1231d69634a6","Type":"ContainerStarted","Data":"b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb"} Oct 31 21:36:18.262019 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.261553 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" event={"ID":"ad33938e-2fbb-47f7-ae05-1231d69634a6","Type":"ContainerStarted","Data":"b752f11db3748b9abf4893f30831004c469571d823c64917047d8782b2b2bbac"} Oct 31 21:36:18.262019 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.261579 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:18.263386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.263349 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" event={"ID":"f265ffe5-40ed-409b-8358-a0ec7732c55f","Type":"ContainerStarted","Data":"66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df"} Oct 31 21:36:18.263510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.263410 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:18.263510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.263427 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" event={"ID":"f265ffe5-40ed-409b-8358-a0ec7732c55f","Type":"ContainerStarted","Data":"e3f74b1c98dc537283faa937b25392a92eb109f7c6f53fad5f2dcc7ff4f76cd0"} Oct 31 21:36:18.349027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.348965 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" podStartSLOduration=2.348948505 podStartE2EDuration="2.348948505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:18.308031786 +0000 UTC m=+1218.433139016" watchObservedRunningTime="2026-10-31 21:36:18.348948505 +0000 UTC m=+1218.474055736" Oct 31 21:36:18.352830 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.352736 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" podStartSLOduration=2.352724777 podStartE2EDuration="2.352724777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:18.349483787 +0000 UTC m=+1218.474591019" watchObservedRunningTime="2026-10-31 21:36:18.352724777 +0000 UTC m=+1218.477832009" Oct 31 21:36:18.998164 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:18.998127 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s"] Oct 31 21:36:19.003336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.003294 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.022057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.022026 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s"] Oct 31 21:36:19.025741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.025705 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-084d807f-serving-cert\"" Oct 31 21:36:19.060916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.060870 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl98n\" (UniqueName: \"kubernetes.io/projected/280ddbf6-14ef-4aef-920d-b411919fdb61-kube-api-access-xl98n\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.061086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.060926 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ddbf6-14ef-4aef-920d-b411919fdb61-openshift-service-ca\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.061158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.061106 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.061158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.061143 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-metrics-certs\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.163009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.162971 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl98n\" (UniqueName: \"kubernetes.io/projected/280ddbf6-14ef-4aef-920d-b411919fdb61-kube-api-access-xl98n\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.163199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.163027 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ddbf6-14ef-4aef-920d-b411919fdb61-openshift-service-ca\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.163199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.163133 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.163199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.163171 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-metrics-certs\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.163438 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:19.163270 4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-serving-cert: secret "maas-api-e2e-mt-084d807f-serving-cert" not found Oct 31 21:36:19.163438 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:19.163365 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls podName:280ddbf6-14ef-4aef-920d-b411919fdb61 nodeName:}" failed. No retries permitted until 2026-10-31 21:36:19.663342497 +0000 UTC m=+1219.788449719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls") pod "maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" (UID: "280ddbf6-14ef-4aef-920d-b411919fdb61") : secret "maas-api-e2e-mt-084d807f-serving-cert" not found Oct 31 21:36:19.163799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.163778 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ddbf6-14ef-4aef-920d-b411919fdb61-openshift-service-ca\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.165889 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.165860 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-metrics-certs\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.224002 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.223723 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl98n\" (UniqueName: \"kubernetes.io/projected/280ddbf6-14ef-4aef-920d-b411919fdb61-kube-api-access-xl98n\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.671772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.671737 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.675138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.675108 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls\") pod \"maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:19.918901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:19.918851 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:20.282430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:20.282402 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s"] Oct 31 21:36:20.283253 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:36:20.283223 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280ddbf6_14ef_4aef_920d_b411919fdb61.slice/crio-ef91e2977f6afd3576c8aaf676744df7c5b354471a868c7f569a0c7fe72ff9e9 WatchSource:0}: Error finding container ef91e2977f6afd3576c8aaf676744df7c5b354471a868c7f569a0c7fe72ff9e9: Status 404 returned error can't find the container with id ef91e2977f6afd3576c8aaf676744df7c5b354471a868c7f569a0c7fe72ff9e9 Oct 31 21:36:21.294521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:21.294472 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" event={"ID":"280ddbf6-14ef-4aef-920d-b411919fdb61","Type":"ContainerStarted","Data":"6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7"} Oct 31 21:36:21.295012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:21.294562 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" event={"ID":"280ddbf6-14ef-4aef-920d-b411919fdb61","Type":"ContainerStarted","Data":"ef91e2977f6afd3576c8aaf676744df7c5b354471a868c7f569a0c7fe72ff9e9"} Oct 31 21:36:21.295012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:21.294585 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:21.340652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:21.340605 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" podStartSLOduration=3.04133333 podStartE2EDuration="3.340592561s" totalImagesPullingTime="299.259231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:21.337634356 +0000 UTC m=+1221.462741588" watchObservedRunningTime="2026-10-31 21:36:21.340592561 +0000 UTC m=+1221.465699792" Oct 31 21:36:27.305998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:27.305967 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:36:27.599228 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:27.599113 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl"] Oct 31 21:36:27.599416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:27.599382 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" podUID="a6070ebd-cd3c-4d68-860b-3f7075efe67e" containerName="maas-api" containerID="cri-o://80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5" gracePeriod=30 Oct 31 21:36:27.956709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:27.956678 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:36:28.105187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.105147 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6070ebd-cd3c-4d68-860b-3f7075efe67e-kube-api-access-n4t5c\" (UniqueName: \"kubernetes.io/projected/a6070ebd-cd3c-4d68-860b-3f7075efe67e-kube-api-access-n4t5c\") pod \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " Oct 31 21:36:28.105429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.105297 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6070ebd-cd3c-4d68-860b-3f7075efe67e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6070ebd-cd3c-4d68-860b-3f7075efe67e-openshift-service-ca\") pod \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " Oct 31 21:36:28.105571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.105453 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls\") pod \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " Oct 31 21:36:28.105661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.105643 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-metrics-certs\") pod \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\" (UID: \"a6070ebd-cd3c-4d68-860b-3f7075efe67e\") " Oct 31 21:36:28.107096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.107068 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6070ebd-cd3c-4d68-860b-3f7075efe67e-openshift-service-ca" pod "a6070ebd-cd3c-4d68-860b-3f7075efe67e" (UID: "a6070ebd-cd3c-4d68-860b-3f7075efe67e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:28.109377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.109338 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls" pod "a6070ebd-cd3c-4d68-860b-3f7075efe67e" (UID: "a6070ebd-cd3c-4d68-860b-3f7075efe67e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:28.111984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.111947 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6070ebd-cd3c-4d68-860b-3f7075efe67e-kube-api-access-n4t5c" pod "a6070ebd-cd3c-4d68-860b-3f7075efe67e" (UID: "a6070ebd-cd3c-4d68-860b-3f7075efe67e"). InnerVolumeSpecName "kube-api-access-n4t5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:28.112810 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.112602 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-metrics-certs" pod "a6070ebd-cd3c-4d68-860b-3f7075efe67e" (UID: "a6070ebd-cd3c-4d68-860b-3f7075efe67e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:36:28.207320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.207223 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:28.207320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.207251 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4t5c\" (UniqueName: \"kubernetes.io/projected/a6070ebd-cd3c-4d68-860b-3f7075efe67e-kube-api-access-n4t5c\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:28.207320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.207269 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6070ebd-cd3c-4d68-860b-3f7075efe67e-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:28.207320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.207278 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6070ebd-cd3c-4d68-860b-3f7075efe67e-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:28.343432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.343396 4294 generic.go:356] "Generic (PLEG): container finished" podID="a6070ebd-cd3c-4d68-860b-3f7075efe67e" containerID="80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5" exitCode=0 Oct 31 21:36:28.343979 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.343483 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" Oct 31 21:36:28.343979 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.343553 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" event={"ID":"a6070ebd-cd3c-4d68-860b-3f7075efe67e","Type":"ContainerDied","Data":"80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5"} Oct 31 21:36:28.343979 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.343610 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl" event={"ID":"a6070ebd-cd3c-4d68-860b-3f7075efe67e","Type":"ContainerDied","Data":"1ef451bfe2ced9aeaf31d0b736d2f3260cb074808683a8d8e2bb181dc6a61ed8"} Oct 31 21:36:28.343979 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.343638 4294 scope.go:122] "RemoveContainer" containerID="80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5" Oct 31 21:36:28.361158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.361137 4294 scope.go:122] "RemoveContainer" containerID="80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5" Oct 31 21:36:28.361575 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:28.361544 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5\": container with ID starting with 80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5 not found: ID does not exist" containerID="80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5" Oct 31 21:36:28.361667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.361592 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5"} err="failed to get container status \"80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5\": rpc error: code = NotFound desc = could not find container \"80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5\": container with ID starting with 80bd29c139bba01345488972f484929c83f00ba5d9554045e7160ac7faed2de5 not found: ID does not exist" Oct 31 21:36:28.399466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.399430 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl"] Oct 31 21:36:28.411566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:28.411537 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-248b29-c94f76d74-ll7tl"] Oct 31 21:36:29.276114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:29.276078 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:36:29.276345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:29.276163 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:36:29.987177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:29.987143 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6070ebd-cd3c-4d68-860b-3f7075efe67e" path="/var/lib/kubelet/pods/a6070ebd-cd3c-4d68-860b-3f7075efe67e/volumes" Oct 31 21:36:31.315049 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:31.314997 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:36:32.387177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:32.387126 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796"] Oct 31 21:36:37.387986 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.387943 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" podUID="19617a06-32d8-4546-9bec-c44088963e6c" containerName="payload-processing" containerID="cri-o://856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36" gracePeriod=25 Oct 31 21:36:37.689770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.689739 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:36:37.762352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.762316 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19617a06-32d8-4546-9bec-c44088963e6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19617a06-32d8-4546-9bec-c44088963e6c-plugins-config-volume\") pod \"19617a06-32d8-4546-9bec-c44088963e6c\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " Oct 31 21:36:37.762574 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.762555 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19617a06-32d8-4546-9bec-c44088963e6c-kube-api-access-zswwg\" (UniqueName: \"kubernetes.io/projected/19617a06-32d8-4546-9bec-c44088963e6c-kube-api-access-zswwg\") pod \"19617a06-32d8-4546-9bec-c44088963e6c\" (UID: \"19617a06-32d8-4546-9bec-c44088963e6c\") " Oct 31 21:36:37.762688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.762662 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19617a06-32d8-4546-9bec-c44088963e6c-plugins-config-volume" pod "19617a06-32d8-4546-9bec-c44088963e6c" (UID: "19617a06-32d8-4546-9bec-c44088963e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:36:37.763118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.763090 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19617a06-32d8-4546-9bec-c44088963e6c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:37.764693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.764659 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19617a06-32d8-4546-9bec-c44088963e6c-kube-api-access-zswwg" pod "19617a06-32d8-4546-9bec-c44088963e6c" (UID: "19617a06-32d8-4546-9bec-c44088963e6c"). InnerVolumeSpecName "kube-api-access-zswwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:37.864883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:37.864845 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswwg\" (UniqueName: \"kubernetes.io/projected/19617a06-32d8-4546-9bec-c44088963e6c-kube-api-access-zswwg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:38.428916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.428832 4294 generic.go:356] "Generic (PLEG): container finished" podID="19617a06-32d8-4546-9bec-c44088963e6c" containerID="856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36" exitCode=0 Oct 31 21:36:38.428916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.428875 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" Oct 31 21:36:38.429437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.428920 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" event={"ID":"19617a06-32d8-4546-9bec-c44088963e6c","Type":"ContainerDied","Data":"856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36"} Oct 31 21:36:38.429437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.428969 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796" event={"ID":"19617a06-32d8-4546-9bec-c44088963e6c","Type":"ContainerDied","Data":"32e2daccb5992e5a050e406b4b601c6b7a26485a67018ecacb3035e3afb9d758"} Oct 31 21:36:38.429437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.428996 4294 scope.go:122] "RemoveContainer" containerID="856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36" Oct 31 21:36:38.438534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.438510 4294 scope.go:122] "RemoveContainer" containerID="856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36" Oct 31 21:36:38.438782 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:38.438764 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36\": container with ID starting with 856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36 not found: ID does not exist" containerID="856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36" Oct 31 21:36:38.438821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.438790 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36"} err="failed to get container status \"856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36\": rpc error: code = NotFound desc = could not find container \"856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36\": container with ID starting with 856f7cdd60b1327263a8d9a4237f5f43996c17b57039a45ae4876c774a86ca36 not found: ID does not exist" Oct 31 21:36:38.486718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.486684 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796"] Oct 31 21:36:38.496664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:38.496637 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-08ddf0-5f75dbd4bc-hs796"] Oct 31 21:36:39.985910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:39.985876 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19617a06-32d8-4546-9bec-c44088963e6c" path="/var/lib/kubelet/pods/19617a06-32d8-4546-9bec-c44088963e6c/volumes" Oct 31 21:36:43.373688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.373648 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7549-9220ab"] Oct 31 21:36:43.374448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.374428 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a6070ebd-cd3c-4d68-860b-3f7075efe67e" containerName="maas-api" Oct 31 21:36:43.374448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.374448 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6070ebd-cd3c-4d68-860b-3f7075efe67e" containerName="maas-api" Oct 31 21:36:43.374619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.374471 4294 cpu_manager.go:414] "containerMap: removing container" podUID="19617a06-32d8-4546-9bec-c44088963e6c" containerName="payload-processing" Oct 31 21:36:43.374619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.374479 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19617a06-32d8-4546-9bec-c44088963e6c" containerName="payload-processing" Oct 31 21:36:43.374716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.374652 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6070ebd-cd3c-4d68-860b-3f7075efe67e" containerName="maas-api" Oct 31 21:36:43.374716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.374667 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="19617a06-32d8-4546-9bec-c44088963e6c" containerName="payload-processing" Oct 31 21:36:43.379251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.379217 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:43.397330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.396388 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-9220ab"] Oct 31 21:36:43.414711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.412628 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dsnkq\"" Oct 31 21:36:43.456400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.456366 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd2zx\" (UniqueName: \"kubernetes.io/projected/758b2131-6ce1-4e56-846e-152ddcecd7ec-kube-api-access-wd2zx\") pod \"test-curl-7549-9220ab\" (UID: \"758b2131-6ce1-4e56-846e-152ddcecd7ec\") " pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:43.557612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.557578 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd2zx\" (UniqueName: \"kubernetes.io/projected/758b2131-6ce1-4e56-846e-152ddcecd7ec-kube-api-access-wd2zx\") pod \"test-curl-7549-9220ab\" (UID: \"758b2131-6ce1-4e56-846e-152ddcecd7ec\") " pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:43.583745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.583711 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd2zx\" (UniqueName: \"kubernetes.io/projected/758b2131-6ce1-4e56-846e-152ddcecd7ec-kube-api-access-wd2zx\") pod \"test-curl-7549-9220ab\" (UID: \"758b2131-6ce1-4e56-846e-152ddcecd7ec\") " pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:43.695384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.695296 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:43.999057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:43.999031 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7549-9220ab"] Oct 31 21:36:44.000952 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:36:44.000925 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod758b2131_6ce1_4e56_846e_152ddcecd7ec.slice/crio-14f4bdbe067ee212880384ee89b90ba4a76d113d9a67151ec46300589e4637eb WatchSource:0}: Error finding container 14f4bdbe067ee212880384ee89b90ba4a76d113d9a67151ec46300589e4637eb: Status 404 returned error can't find the container with id 14f4bdbe067ee212880384ee89b90ba4a76d113d9a67151ec46300589e4637eb Oct 31 21:36:44.471416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:44.471329 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-9220ab" event={"ID":"758b2131-6ce1-4e56-846e-152ddcecd7ec","Type":"ContainerStarted","Data":"386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7"} Oct 31 21:36:44.471416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:44.471373 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7549-9220ab" event={"ID":"758b2131-6ce1-4e56-846e-152ddcecd7ec","Type":"ContainerStarted","Data":"14f4bdbe067ee212880384ee89b90ba4a76d113d9a67151ec46300589e4637eb"} Oct 31 21:36:44.510388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:44.510341 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7549-9220ab" podStartSLOduration=1.510325564 podStartE2EDuration="1.510325564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:44.509118704 +0000 UTC m=+1244.634225931" watchObservedRunningTime="2026-10-31 21:36:44.510325564 +0000 UTC m=+1244.635432795" Oct 31 21:36:45.392363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:45.392323 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7549-9220ab"] Oct 31 21:36:45.399203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:45.399171 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7549-9220ab"] Oct 31 21:36:46.493125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:46.493077 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7549-9220ab" podUID="758b2131-6ce1-4e56-846e-152ddcecd7ec" containerName="test-curl-7549-9220ab" containerID="cri-o://386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7" gracePeriod=2 Oct 31 21:36:47.386586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.386554 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw"] Oct 31 21:36:47.387176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.387163 4294 cpu_manager.go:414] "containerMap: removing container" podUID="758b2131-6ce1-4e56-846e-152ddcecd7ec" containerName="test-curl-7549-9220ab" Oct 31 21:36:47.387228 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.387178 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="758b2131-6ce1-4e56-846e-152ddcecd7ec" containerName="test-curl-7549-9220ab" Oct 31 21:36:47.387367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.387356 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="758b2131-6ce1-4e56-846e-152ddcecd7ec" containerName="test-curl-7549-9220ab" Oct 31 21:36:47.390496 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.390480 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.409849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.409820 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw"] Oct 31 21:36:47.418441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.418400 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f1e791-dockercfg-xn5tl\"" Oct 31 21:36:47.418441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.418416 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f1e791\"" Oct 31 21:36:47.549331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.534687 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mmn\" (UniqueName: \"kubernetes.io/projected/04ac6d67-3ecb-4c43-9df3-b2a270278829-kube-api-access-r7mmn\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.549928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.549896 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ac6d67-3ecb-4c43-9df3-b2a270278829-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.600444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.600402 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf"] Oct 31 21:36:47.638932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.638844 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf"] Oct 31 21:36:47.639114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.638997 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:47.661504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.661472 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rtd\" (UniqueName: \"kubernetes.io/projected/b3972990-9501-4a33-97b9-84df70dcd649-kube-api-access-k6rtd\") pod \"payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:47.661817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.661794 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mmn\" (UniqueName: \"kubernetes.io/projected/04ac6d67-3ecb-4c43-9df3-b2a270278829-kube-api-access-r7mmn\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.661958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.661943 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3972990-9501-4a33-97b9-84df70dcd649-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:47.662093 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.662078 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ac6d67-3ecb-4c43-9df3-b2a270278829-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.663225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.663199 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ac6d67-3ecb-4c43-9df3-b2a270278829-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.688490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.688444 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mmn\" (UniqueName: \"kubernetes.io/projected/04ac6d67-3ecb-4c43-9df3-b2a270278829-kube-api-access-r7mmn\") pod \"payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.704397 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.704335 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:47.764840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.763642 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rtd\" (UniqueName: \"kubernetes.io/projected/b3972990-9501-4a33-97b9-84df70dcd649-kube-api-access-k6rtd\") pod \"payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:47.764840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.763723 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3972990-9501-4a33-97b9-84df70dcd649-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:47.764840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.764533 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3972990-9501-4a33-97b9-84df70dcd649-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:47.791001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.790939 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rtd\" (UniqueName: \"kubernetes.io/projected/b3972990-9501-4a33-97b9-84df70dcd649-kube-api-access-k6rtd\") pod \"payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:47.973580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:47.973503 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:48.040819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.040784 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw"] Oct 31 21:36:48.279599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.279571 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf"] Oct 31 21:36:48.280411 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:36:48.280383 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3972990_9501_4a33_97b9_84df70dcd649.slice/crio-9cc26e762785c1b5fa2b84fa0f4196389dee78520de19f17087394b8b8cdadcb WatchSource:0}: Error finding container 9cc26e762785c1b5fa2b84fa0f4196389dee78520de19f17087394b8b8cdadcb: Status 404 returned error can't find the container with id 9cc26e762785c1b5fa2b84fa0f4196389dee78520de19f17087394b8b8cdadcb Oct 31 21:36:48.531296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.531089 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" event={"ID":"04ac6d67-3ecb-4c43-9df3-b2a270278829","Type":"ContainerStarted","Data":"beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c"} Oct 31 21:36:48.531460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.531297 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" event={"ID":"04ac6d67-3ecb-4c43-9df3-b2a270278829","Type":"ContainerStarted","Data":"3cfd45a23c569ec9b144a0308340380f389ba58e1d1cacd2f72e8455b7074963"} Oct 31 21:36:48.531460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.531438 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:48.532807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.532781 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" event={"ID":"b3972990-9501-4a33-97b9-84df70dcd649","Type":"ContainerStarted","Data":"3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca"} Oct 31 21:36:48.532918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.532830 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" event={"ID":"b3972990-9501-4a33-97b9-84df70dcd649","Type":"ContainerStarted","Data":"9cc26e762785c1b5fa2b84fa0f4196389dee78520de19f17087394b8b8cdadcb"} Oct 31 21:36:48.532968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.532916 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:36:48.576208 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.576160 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" podStartSLOduration=1.576146224 podStartE2EDuration="1.576146224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:48.571796986 +0000 UTC m=+1248.696904218" watchObservedRunningTime="2026-10-31 21:36:48.576146224 +0000 UTC m=+1248.701253456" Oct 31 21:36:48.619996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.619933 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" podStartSLOduration=1.61991245 podStartE2EDuration="1.61991245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:36:48.615404788 +0000 UTC m=+1248.740512021" watchObservedRunningTime="2026-10-31 21:36:48.61991245 +0000 UTC m=+1248.745019682" Oct 31 21:36:48.681486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.681461 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:48.779341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.779294 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/758b2131-6ce1-4e56-846e-152ddcecd7ec-kube-api-access-wd2zx\" (UniqueName: \"kubernetes.io/projected/758b2131-6ce1-4e56-846e-152ddcecd7ec-kube-api-access-wd2zx\") pod \"758b2131-6ce1-4e56-846e-152ddcecd7ec\" (UID: \"758b2131-6ce1-4e56-846e-152ddcecd7ec\") " Oct 31 21:36:48.781851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.781780 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/758b2131-6ce1-4e56-846e-152ddcecd7ec-kube-api-access-wd2zx" pod "758b2131-6ce1-4e56-846e-152ddcecd7ec" (UID: "758b2131-6ce1-4e56-846e-152ddcecd7ec"). InnerVolumeSpecName "kube-api-access-wd2zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:36:48.881040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:48.881010 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd2zx\" (UniqueName: \"kubernetes.io/projected/758b2131-6ce1-4e56-846e-152ddcecd7ec-kube-api-access-wd2zx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:36:49.540339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:49.540292 4294 generic.go:356] "Generic (PLEG): container finished" podID="758b2131-6ce1-4e56-846e-152ddcecd7ec" containerID="386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7" exitCode=137 Oct 31 21:36:49.540505 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:49.540351 4294 scope.go:122] "RemoveContainer" containerID="386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7" Oct 31 21:36:49.540505 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:49.540357 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:49.550096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:49.550079 4294 scope.go:122] "RemoveContainer" containerID="386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7" Oct 31 21:36:49.550379 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:49.550359 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7\": container with ID starting with 386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7 not found: ID does not exist" containerID="386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7" Oct 31 21:36:49.550436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:49.550387 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7"} err="failed to get container status \"386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7\": rpc error: code = NotFound desc = could not find container \"386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7\": container with ID starting with 386b281e9b26f8ad97fbd589520f848ea609139dd3c028dbb8078d120d27c9e7 not found: ID does not exist" Oct 31 21:36:49.562745 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:49.562719 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7549-9220ab\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="758b2131-6ce1-4e56-846e-152ddcecd7ec" pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:49.985084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:49.985057 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="758b2131-6ce1-4e56-846e-152ddcecd7ec" path="/var/lib/kubelet/pods/758b2131-6ce1-4e56-846e-152ddcecd7ec/volumes" Oct 31 21:36:49.994113 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:36:49.994087 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7549-9220ab\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="758b2131-6ce1-4e56-846e-152ddcecd7ec" pod="opendatahub/test-curl-7549-9220ab" Oct 31 21:36:56.326653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:56.326619 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r"] Oct 31 21:36:56.355056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:56.349545 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx"] Oct 31 21:36:56.402419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:56.402382 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:36:56.429859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:56.429393 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:36:59.543991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:59.543956 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 21:36:59.544481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:36:59.544104 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 21:37:01.310496 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:01.310455 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:37:01.327670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.327627 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" podUID="f265ffe5-40ed-409b-8358-a0ec7732c55f" containerName="payload-processing" containerID="cri-o://66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df" gracePeriod=25 Oct 31 21:37:01.350159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.350127 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" podUID="ad33938e-2fbb-47f7-ae05-1231d69634a6" containerName="payload-pre-processing" containerID="cri-o://b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb" gracePeriod=25 Oct 31 21:37:01.622696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.622668 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:37:01.685190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.685159 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f265ffe5-40ed-409b-8358-a0ec7732c55f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f265ffe5-40ed-409b-8358-a0ec7732c55f-plugins-config-volume\") pod \"f265ffe5-40ed-409b-8358-a0ec7732c55f\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " Oct 31 21:37:01.685544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.685523 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f265ffe5-40ed-409b-8358-a0ec7732c55f-kube-api-access-mbc8b\" (UniqueName: \"kubernetes.io/projected/f265ffe5-40ed-409b-8358-a0ec7732c55f-kube-api-access-mbc8b\") pod \"f265ffe5-40ed-409b-8358-a0ec7732c55f\" (UID: \"f265ffe5-40ed-409b-8358-a0ec7732c55f\") " Oct 31 21:37:01.685651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.685579 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f265ffe5-40ed-409b-8358-a0ec7732c55f-plugins-config-volume" pod "f265ffe5-40ed-409b-8358-a0ec7732c55f" (UID: "f265ffe5-40ed-409b-8358-a0ec7732c55f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:37:01.686411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.686128 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f265ffe5-40ed-409b-8358-a0ec7732c55f-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:01.688259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.688231 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f265ffe5-40ed-409b-8358-a0ec7732c55f-kube-api-access-mbc8b" pod "f265ffe5-40ed-409b-8358-a0ec7732c55f" (UID: "f265ffe5-40ed-409b-8358-a0ec7732c55f"). InnerVolumeSpecName "kube-api-access-mbc8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:37:01.691135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.691115 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:37:01.711442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.711413 4294 generic.go:356] "Generic (PLEG): container finished" podID="f265ffe5-40ed-409b-8358-a0ec7732c55f" containerID="66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df" exitCode=0 Oct 31 21:37:01.711569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.711458 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" Oct 31 21:37:01.711569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.711493 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" event={"ID":"f265ffe5-40ed-409b-8358-a0ec7732c55f","Type":"ContainerDied","Data":"66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df"} Oct 31 21:37:01.711569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.711561 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r" event={"ID":"f265ffe5-40ed-409b-8358-a0ec7732c55f","Type":"ContainerDied","Data":"e3f74b1c98dc537283faa937b25392a92eb109f7c6f53fad5f2dcc7ff4f76cd0"} Oct 31 21:37:01.711701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.711585 4294 scope.go:122] "RemoveContainer" containerID="66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df" Oct 31 21:37:01.715144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.715073 4294 generic.go:356] "Generic (PLEG): container finished" podID="ad33938e-2fbb-47f7-ae05-1231d69634a6" containerID="b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb" exitCode=0 Oct 31 21:37:01.715144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.715098 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" event={"ID":"ad33938e-2fbb-47f7-ae05-1231d69634a6","Type":"ContainerDied","Data":"b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb"} Oct 31 21:37:01.715144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.715131 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" Oct 31 21:37:01.715387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.715162 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx" event={"ID":"ad33938e-2fbb-47f7-ae05-1231d69634a6","Type":"ContainerDied","Data":"b752f11db3748b9abf4893f30831004c469571d823c64917047d8782b2b2bbac"} Oct 31 21:37:01.724499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.724481 4294 scope.go:122] "RemoveContainer" containerID="66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df" Oct 31 21:37:01.724810 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:01.724789 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df\": container with ID starting with 66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df not found: ID does not exist" containerID="66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df" Oct 31 21:37:01.724888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.724817 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df"} err="failed to get container status \"66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df\": rpc error: code = NotFound desc = could not find container \"66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df\": container with ID starting with 66db8ca8147041a87de2d45cb477bb031b0e87286a047494aa8c08a3e58267df not found: ID does not exist" Oct 31 21:37:01.724888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.724836 4294 scope.go:122] "RemoveContainer" containerID="b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb" Oct 31 21:37:01.745563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.745541 4294 scope.go:122] "RemoveContainer" containerID="b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb" Oct 31 21:37:01.745886 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:01.745861 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb\": container with ID starting with b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb not found: ID does not exist" containerID="b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb" Oct 31 21:37:01.745986 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.745896 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb"} err="failed to get container status \"b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb\": rpc error: code = NotFound desc = could not find container \"b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb\": container with ID starting with b00f3d84d53e5dcae346b4c6842170865b5f57c8035f85d5fb87851bb013bfbb not found: ID does not exist" Oct 31 21:37:01.781359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.781325 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r"] Oct 31 21:37:01.786613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.786583 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-084d807f-7648554b69-4gw5r"] Oct 31 21:37:01.786902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.786882 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad33938e-2fbb-47f7-ae05-1231d69634a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad33938e-2fbb-47f7-ae05-1231d69634a6-plugins-config-volume\") pod \"ad33938e-2fbb-47f7-ae05-1231d69634a6\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " Oct 31 21:37:01.787124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.787108 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad33938e-2fbb-47f7-ae05-1231d69634a6-kube-api-access-c6kpk\" (UniqueName: \"kubernetes.io/projected/ad33938e-2fbb-47f7-ae05-1231d69634a6-kube-api-access-c6kpk\") pod \"ad33938e-2fbb-47f7-ae05-1231d69634a6\" (UID: \"ad33938e-2fbb-47f7-ae05-1231d69634a6\") " Oct 31 21:37:01.787257 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.787237 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad33938e-2fbb-47f7-ae05-1231d69634a6-plugins-config-volume" pod "ad33938e-2fbb-47f7-ae05-1231d69634a6" (UID: "ad33938e-2fbb-47f7-ae05-1231d69634a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:37:01.787586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.787558 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbc8b\" (UniqueName: \"kubernetes.io/projected/f265ffe5-40ed-409b-8358-a0ec7732c55f-kube-api-access-mbc8b\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:01.787586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.787578 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad33938e-2fbb-47f7-ae05-1231d69634a6-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:01.789045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.789026 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad33938e-2fbb-47f7-ae05-1231d69634a6-kube-api-access-c6kpk" pod "ad33938e-2fbb-47f7-ae05-1231d69634a6" (UID: "ad33938e-2fbb-47f7-ae05-1231d69634a6"). InnerVolumeSpecName "kube-api-access-c6kpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:37:01.890940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.889789 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6kpk\" (UniqueName: \"kubernetes.io/projected/ad33938e-2fbb-47f7-ae05-1231d69634a6-kube-api-access-c6kpk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:01.986216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:01.986183 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f265ffe5-40ed-409b-8358-a0ec7732c55f" path="/var/lib/kubelet/pods/f265ffe5-40ed-409b-8358-a0ec7732c55f/volumes" Oct 31 21:37:02.057820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:02.057790 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx"] Oct 31 21:37:02.061944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:02.061920 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-084d807f-5ff89dd6bc-q72sx"] Oct 31 21:37:03.985491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:03.985457 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad33938e-2fbb-47f7-ae05-1231d69634a6" path="/var/lib/kubelet/pods/ad33938e-2fbb-47f7-ae05-1231d69634a6/volumes" Oct 31 21:37:08.670099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:08.670061 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s"] Oct 31 21:37:08.670574 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:08.670400 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" podUID="280ddbf6-14ef-4aef-920d-b411919fdb61" containerName="maas-api" containerID="cri-o://6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7" gracePeriod=30 Oct 31 21:37:09.076758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.076294 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:37:09.200177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.200129 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280ddbf6-14ef-4aef-920d-b411919fdb61-kube-api-access-xl98n\" (UniqueName: \"kubernetes.io/projected/280ddbf6-14ef-4aef-920d-b411919fdb61-kube-api-access-xl98n\") pod \"280ddbf6-14ef-4aef-920d-b411919fdb61\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " Oct 31 21:37:09.200379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.200361 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-metrics-certs\") pod \"280ddbf6-14ef-4aef-920d-b411919fdb61\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " Oct 31 21:37:09.200437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.200402 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls\") pod \"280ddbf6-14ef-4aef-920d-b411919fdb61\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " Oct 31 21:37:09.200589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.200569 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280ddbf6-14ef-4aef-920d-b411919fdb61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ddbf6-14ef-4aef-920d-b411919fdb61-openshift-service-ca\") pod \"280ddbf6-14ef-4aef-920d-b411919fdb61\" (UID: \"280ddbf6-14ef-4aef-920d-b411919fdb61\") " Oct 31 21:37:09.201005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.200973 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ddbf6-14ef-4aef-920d-b411919fdb61-openshift-service-ca" pod "280ddbf6-14ef-4aef-920d-b411919fdb61" (UID: "280ddbf6-14ef-4aef-920d-b411919fdb61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:37:09.201489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.201460 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/280ddbf6-14ef-4aef-920d-b411919fdb61-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:09.202785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.202758 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-metrics-certs" pod "280ddbf6-14ef-4aef-920d-b411919fdb61" (UID: "280ddbf6-14ef-4aef-920d-b411919fdb61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:37:09.207911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.207889 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280ddbf6-14ef-4aef-920d-b411919fdb61-kube-api-access-xl98n" pod "280ddbf6-14ef-4aef-920d-b411919fdb61" (UID: "280ddbf6-14ef-4aef-920d-b411919fdb61"). InnerVolumeSpecName "kube-api-access-xl98n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:37:09.208683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.208652 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls" pod "280ddbf6-14ef-4aef-920d-b411919fdb61" (UID: "280ddbf6-14ef-4aef-920d-b411919fdb61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:37:09.303433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.303333 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl98n\" (UniqueName: \"kubernetes.io/projected/280ddbf6-14ef-4aef-920d-b411919fdb61-kube-api-access-xl98n\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:09.303433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.303373 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:09.303433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.303385 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/280ddbf6-14ef-4aef-920d-b411919fdb61-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:09.768158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.768122 4294 generic.go:356] "Generic (PLEG): container finished" podID="280ddbf6-14ef-4aef-920d-b411919fdb61" containerID="6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7" exitCode=0 Oct 31 21:37:09.768666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.768194 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" Oct 31 21:37:09.768666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.768213 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" event={"ID":"280ddbf6-14ef-4aef-920d-b411919fdb61","Type":"ContainerDied","Data":"6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7"} Oct 31 21:37:09.768666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.768359 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s" event={"ID":"280ddbf6-14ef-4aef-920d-b411919fdb61","Type":"ContainerDied","Data":"ef91e2977f6afd3576c8aaf676744df7c5b354471a868c7f569a0c7fe72ff9e9"} Oct 31 21:37:09.768666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.768393 4294 scope.go:122] "RemoveContainer" containerID="6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7" Oct 31 21:37:09.792533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.792502 4294 scope.go:122] "RemoveContainer" containerID="6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7" Oct 31 21:37:09.793162 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:09.793138 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7\": container with ID starting with 6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7 not found: ID does not exist" containerID="6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7" Oct 31 21:37:09.793295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.793169 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7"} err="failed to get container status \"6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7\": rpc error: code = NotFound desc = could not find container \"6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7\": container with ID starting with 6cb625eedd64a95db44188699de84eeb1255d0019442c027be52916239638af7 not found: ID does not exist" Oct 31 21:37:09.833756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.833725 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s"] Oct 31 21:37:09.844976 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.844952 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-084d807f-5dd4bfbb9b-75h2s"] Oct 31 21:37:09.985091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:09.985053 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280ddbf6-14ef-4aef-920d-b411919fdb61" path="/var/lib/kubelet/pods/280ddbf6-14ef-4aef-920d-b411919fdb61/volumes" Oct 31 21:37:14.966065 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:14.966024 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg"] Oct 31 21:37:16.243151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:16.243114 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7"] Oct 31 21:37:21.243743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.243680 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" podUID="91fc09a6-3733-4d20-a197-30304074e9ce" containerName="payload-processing" containerID="cri-o://81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b" gracePeriod=25 Oct 31 21:37:21.515148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.515118 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:37:21.593508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.593465 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fc09a6-3733-4d20-a197-30304074e9ce-kube-api-access-l5zrr\" (UniqueName: \"kubernetes.io/projected/91fc09a6-3733-4d20-a197-30304074e9ce-kube-api-access-l5zrr\") pod \"91fc09a6-3733-4d20-a197-30304074e9ce\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " Oct 31 21:37:21.593700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.593535 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fc09a6-3733-4d20-a197-30304074e9ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fc09a6-3733-4d20-a197-30304074e9ce-plugins-config-volume\") pod \"91fc09a6-3733-4d20-a197-30304074e9ce\" (UID: \"91fc09a6-3733-4d20-a197-30304074e9ce\") " Oct 31 21:37:21.593956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.593934 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fc09a6-3733-4d20-a197-30304074e9ce-plugins-config-volume" pod "91fc09a6-3733-4d20-a197-30304074e9ce" (UID: "91fc09a6-3733-4d20-a197-30304074e9ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:37:21.594160 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.594142 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91fc09a6-3733-4d20-a197-30304074e9ce-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:21.596172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.596149 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fc09a6-3733-4d20-a197-30304074e9ce-kube-api-access-l5zrr" pod "91fc09a6-3733-4d20-a197-30304074e9ce" (UID: "91fc09a6-3733-4d20-a197-30304074e9ce"). InnerVolumeSpecName "kube-api-access-l5zrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:37:21.695638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.695599 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5zrr\" (UniqueName: \"kubernetes.io/projected/91fc09a6-3733-4d20-a197-30304074e9ce-kube-api-access-l5zrr\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:21.856395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.856257 4294 generic.go:356] "Generic (PLEG): container finished" podID="91fc09a6-3733-4d20-a197-30304074e9ce" containerID="81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b" exitCode=0 Oct 31 21:37:21.856395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.856343 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" Oct 31 21:37:21.856673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.856407 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" event={"ID":"91fc09a6-3733-4d20-a197-30304074e9ce","Type":"ContainerDied","Data":"81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b"} Oct 31 21:37:21.856673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.856456 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7" event={"ID":"91fc09a6-3733-4d20-a197-30304074e9ce","Type":"ContainerDied","Data":"e59fa3f4303241715d22bbac71823540a1f4bd7040f09d8e4fd8440c4d4630b0"} Oct 31 21:37:21.856673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.856484 4294 scope.go:122] "RemoveContainer" containerID="81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b" Oct 31 21:37:21.886445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.886344 4294 scope.go:122] "RemoveContainer" containerID="81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b" Oct 31 21:37:21.886812 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:21.886740 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b\": container with ID starting with 81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b not found: ID does not exist" containerID="81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b" Oct 31 21:37:21.886812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.886785 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b"} err="failed to get container status \"81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b\": rpc error: code = NotFound desc = could not find container \"81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b\": container with ID starting with 81be3351dfdc3a72dcead06189e69cb6d0c36b753693792233cb4aa9358b553b not found: ID does not exist" Oct 31 21:37:21.925487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.925458 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7"] Oct 31 21:37:21.932274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:21.932235 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-43fb1b-78b54d5df-9nst7"] Oct 31 21:37:22.061343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:22.056728 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fc09a6-3733-4d20-a197-30304074e9ce" path="/var/lib/kubelet/pods/91fc09a6-3733-4d20-a197-30304074e9ce/volumes" Oct 31 21:37:27.963243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963209 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6"] Oct 31 21:37:27.963791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963776 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f265ffe5-40ed-409b-8358-a0ec7732c55f" containerName="payload-processing" Oct 31 21:37:27.963833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963792 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f265ffe5-40ed-409b-8358-a0ec7732c55f" containerName="payload-processing" Oct 31 21:37:27.963833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963801 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ad33938e-2fbb-47f7-ae05-1231d69634a6" containerName="payload-pre-processing" Oct 31 21:37:27.963833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963806 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad33938e-2fbb-47f7-ae05-1231d69634a6" containerName="payload-pre-processing" Oct 31 21:37:27.963833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963827 4294 cpu_manager.go:414] "containerMap: removing container" podUID="91fc09a6-3733-4d20-a197-30304074e9ce" containerName="payload-processing" Oct 31 21:37:27.963833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963832 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fc09a6-3733-4d20-a197-30304074e9ce" containerName="payload-processing" Oct 31 21:37:27.963989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963841 4294 cpu_manager.go:414] "containerMap: removing container" podUID="280ddbf6-14ef-4aef-920d-b411919fdb61" containerName="maas-api" Oct 31 21:37:27.963989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963846 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280ddbf6-14ef-4aef-920d-b411919fdb61" containerName="maas-api" Oct 31 21:37:27.963989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963928 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fc09a6-3733-4d20-a197-30304074e9ce" containerName="payload-processing" Oct 31 21:37:27.963989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963935 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="280ddbf6-14ef-4aef-920d-b411919fdb61" containerName="maas-api" Oct 31 21:37:27.963989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963946 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f265ffe5-40ed-409b-8358-a0ec7732c55f" containerName="payload-processing" Oct 31 21:37:27.963989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.963953 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad33938e-2fbb-47f7-ae05-1231d69634a6" containerName="payload-pre-processing" Oct 31 21:37:27.968803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.968781 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:27.987479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.987453 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6"] Oct 31 21:37:27.995460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.995436 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-c744c0\"" Oct 31 21:37:27.995594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:27.995544 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-c744c0-dockercfg-b5vkf\"" Oct 31 21:37:28.072718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.072683 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6"] Oct 31 21:37:28.096354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.080809 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.096354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.084010 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6"] Oct 31 21:37:28.096354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.094927 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlspk\" (UniqueName: \"kubernetes.io/projected/97f1ea75-483a-44a8-aab9-ce156938abe5-kube-api-access-nlspk\") pod \"payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.096354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.095092 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f1ea75-483a-44a8-aab9-ce156938abe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.213382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.197363 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wx49\" (UniqueName: \"kubernetes.io/projected/835a2433-6d43-4795-b33f-3a8f3bbf586d-kube-api-access-8wx49\") pod \"payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.213382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.197452 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlspk\" (UniqueName: \"kubernetes.io/projected/97f1ea75-483a-44a8-aab9-ce156938abe5-kube-api-access-nlspk\") pod \"payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.213382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.197597 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835a2433-6d43-4795-b33f-3a8f3bbf586d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.213382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.197673 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f1ea75-483a-44a8-aab9-ce156938abe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.213382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.198740 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f1ea75-483a-44a8-aab9-ce156938abe5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.222416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.222388 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlspk\" (UniqueName: \"kubernetes.io/projected/97f1ea75-483a-44a8-aab9-ce156938abe5-kube-api-access-nlspk\") pod \"payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.282247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.282214 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.298859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.298827 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835a2433-6d43-4795-b33f-3a8f3bbf586d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.299020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.299003 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wx49\" (UniqueName: \"kubernetes.io/projected/835a2433-6d43-4795-b33f-3a8f3bbf586d-kube-api-access-8wx49\") pod \"payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.299540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.299518 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835a2433-6d43-4795-b33f-3a8f3bbf586d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.322955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.322924 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wx49\" (UniqueName: \"kubernetes.io/projected/835a2433-6d43-4795-b33f-3a8f3bbf586d-kube-api-access-8wx49\") pod \"payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.419613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.419576 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.590584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.590558 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6"] Oct 31 21:37:28.591774 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:37:28.591751 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f1ea75_483a_44a8_aab9_ce156938abe5.slice/crio-0fb22bd15e698b6378cf62001ff2eaf2f3ef8c980a25ffd190200888398b8f91 WatchSource:0}: Error finding container 0fb22bd15e698b6378cf62001ff2eaf2f3ef8c980a25ffd190200888398b8f91: Status 404 returned error can't find the container with id 0fb22bd15e698b6378cf62001ff2eaf2f3ef8c980a25ffd190200888398b8f91 Oct 31 21:37:28.724002 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.723979 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6"] Oct 31 21:37:28.725209 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:37:28.725179 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835a2433_6d43_4795_b33f_3a8f3bbf586d.slice/crio-8f7c9cd2df1d7a1f08e23120113bb80bd8e6dd6c35775a8194440172f4be2382 WatchSource:0}: Error finding container 8f7c9cd2df1d7a1f08e23120113bb80bd8e6dd6c35775a8194440172f4be2382: Status 404 returned error can't find the container with id 8f7c9cd2df1d7a1f08e23120113bb80bd8e6dd6c35775a8194440172f4be2382 Oct 31 21:37:28.904882 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.904840 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" event={"ID":"835a2433-6d43-4795-b33f-3a8f3bbf586d","Type":"ContainerStarted","Data":"2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258"} Oct 31 21:37:28.905077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.904909 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" event={"ID":"835a2433-6d43-4795-b33f-3a8f3bbf586d","Type":"ContainerStarted","Data":"8f7c9cd2df1d7a1f08e23120113bb80bd8e6dd6c35775a8194440172f4be2382"} Oct 31 21:37:28.905077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.905024 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:28.906438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.906411 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" event={"ID":"97f1ea75-483a-44a8-aab9-ce156938abe5","Type":"ContainerStarted","Data":"2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd"} Oct 31 21:37:28.906558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.906459 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:28.906558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.906480 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" event={"ID":"97f1ea75-483a-44a8-aab9-ce156938abe5","Type":"ContainerStarted","Data":"0fb22bd15e698b6378cf62001ff2eaf2f3ef8c980a25ffd190200888398b8f91"} Oct 31 21:37:28.947474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.947428 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" podStartSLOduration=0.94741437 podStartE2EDuration="947.41437ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:28.946693411 +0000 UTC m=+1289.071800644" watchObservedRunningTime="2026-10-31 21:37:28.94741437 +0000 UTC m=+1289.072521630" Oct 31 21:37:28.987021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:28.986973 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" podStartSLOduration=1.986958917 podStartE2EDuration="1.986958917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:28.986547385 +0000 UTC m=+1289.111654618" watchObservedRunningTime="2026-10-31 21:37:28.986958917 +0000 UTC m=+1289.112066147" Oct 31 21:37:29.987043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:29.986997 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" podUID="fbe06c24-13f3-482f-9be7-e71f02a3faa1" containerName="main" containerID="cri-o://097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39" gracePeriod=45 Oct 31 21:37:30.261748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.261724 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:37:30.428685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.428653 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-home\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-home\") pod \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " Oct 31 21:37:30.428861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.428710 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-model-cache\") pod \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " Oct 31 21:37:30.428861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.428769 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tmp-dir\") pod \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " Oct 31 21:37:30.428978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.428883 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe06c24-13f3-482f-9be7-e71f02a3faa1-kube-api-access-c4jj4\" (UniqueName: \"kubernetes.io/projected/fbe06c24-13f3-482f-9be7-e71f02a3faa1-kube-api-access-c4jj4\") pod \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " Oct 31 21:37:30.428978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.428930 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tls-certs\") pod \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " Oct 31 21:37:30.428978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.428950 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-home" pod "fbe06c24-13f3-482f-9be7-e71f02a3faa1" (UID: "fbe06c24-13f3-482f-9be7-e71f02a3faa1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:37:30.429145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.429004 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-model-cache" pod "fbe06c24-13f3-482f-9be7-e71f02a3faa1" (UID: "fbe06c24-13f3-482f-9be7-e71f02a3faa1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:37:30.429145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.429028 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-dshm\") pod \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\" (UID: \"fbe06c24-13f3-482f-9be7-e71f02a3faa1\") " Oct 31 21:37:30.429145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.429094 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tmp-dir" pod "fbe06c24-13f3-482f-9be7-e71f02a3faa1" (UID: "fbe06c24-13f3-482f-9be7-e71f02a3faa1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:37:30.429663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.429630 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:30.429663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.429653 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:30.429663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.429666 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:30.431188 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.431161 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tls-certs" pod "fbe06c24-13f3-482f-9be7-e71f02a3faa1" (UID: "fbe06c24-13f3-482f-9be7-e71f02a3faa1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:37:30.431279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.431202 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe06c24-13f3-482f-9be7-e71f02a3faa1-kube-api-access-c4jj4" pod "fbe06c24-13f3-482f-9be7-e71f02a3faa1" (UID: "fbe06c24-13f3-482f-9be7-e71f02a3faa1"). InnerVolumeSpecName "kube-api-access-c4jj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:37:30.431279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.431240 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-dshm" pod "fbe06c24-13f3-482f-9be7-e71f02a3faa1" (UID: "fbe06c24-13f3-482f-9be7-e71f02a3faa1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:37:30.530616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.530527 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4jj4\" (UniqueName: \"kubernetes.io/projected/fbe06c24-13f3-482f-9be7-e71f02a3faa1-kube-api-access-c4jj4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:30.530616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.530562 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbe06c24-13f3-482f-9be7-e71f02a3faa1-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:30.530616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.530574 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbe06c24-13f3-482f-9be7-e71f02a3faa1-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:37:30.919230 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.919135 4294 generic.go:356] "Generic (PLEG): container finished" podID="fbe06c24-13f3-482f-9be7-e71f02a3faa1" containerID="097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39" exitCode=0 Oct 31 21:37:30.919230 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.919196 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" Oct 31 21:37:30.919482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.919221 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" event={"ID":"fbe06c24-13f3-482f-9be7-e71f02a3faa1","Type":"ContainerDied","Data":"097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39"} Oct 31 21:37:30.919482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.919276 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" event={"ID":"fbe06c24-13f3-482f-9be7-e71f02a3faa1","Type":"ContainerDied","Data":"8359b22cbeb11fcefdc9580c8ae2fedc02b8cfd3c50dda998a350bc8c0fe60bf"} Oct 31 21:37:30.919482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.919292 4294 scope.go:122] "RemoveContainer" containerID="097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39" Oct 31 21:37:30.932854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.932835 4294 scope.go:122] "RemoveContainer" containerID="097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39" Oct 31 21:37:30.933125 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:30.933103 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39\": container with ID starting with 097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39 not found: ID does not exist" containerID="097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39" Oct 31 21:37:30.933176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.933141 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39"} err="failed to get container status \"097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39\": rpc error: code = NotFound desc = could not find container \"097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39\": container with ID starting with 097d4ae6017c8d1c02171a82180b2574762b1c7cfd30b2d28705c3ebeef97a39 not found: ID does not exist" Oct 31 21:37:30.988231 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:30.988198 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg"] Oct 31 21:37:31.000138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:31.000111 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg"] Oct 31 21:37:31.179420 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:31.179331 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-084d807f/e2e-lifecycle-model-084d807f-kserve-57547c6cbf-8hkgg" podUID="fbe06c24-13f3-482f-9be7-e71f02a3faa1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:37:31.325027 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:31.324927 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:37:31.985567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:31.985536 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe06c24-13f3-482f-9be7-e71f02a3faa1" path="/var/lib/kubelet/pods/fbe06c24-13f3-482f-9be7-e71f02a3faa1/volumes" Oct 31 21:37:32.245123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.245044 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr"] Oct 31 21:37:32.245643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.245626 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fbe06c24-13f3-482f-9be7-e71f02a3faa1" containerName="main" Oct 31 21:37:32.245643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.245642 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe06c24-13f3-482f-9be7-e71f02a3faa1" containerName="main" Oct 31 21:37:32.245761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.245723 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe06c24-13f3-482f-9be7-e71f02a3faa1" containerName="main" Oct 31 21:37:32.248268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.248246 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.268408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.263682 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-fc0d65-openshift-default-dockercfg-6kfdl\"" Oct 31 21:37:32.282048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.282016 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr"] Oct 31 21:37:32.353757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.353726 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-credential-socket\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.353951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.353792 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-envoy\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.353951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.353863 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-podinfo\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.353951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.353894 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-socket\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.353951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.353940 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-token\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.354195 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.353998 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11df5d62-c1f6-4651-b068-b0bacee10d4a-istiod-ca-cert\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.354195 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.354021 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjhmf\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-kube-api-access-pjhmf\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.354195 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.354045 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-data\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.354195 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.354080 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-certs\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455337 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-credential-socket\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455453 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-envoy\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455538 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-podinfo\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455567 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-socket\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455600 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-token\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455693 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11df5d62-c1f6-4651-b068-b0bacee10d4a-istiod-ca-cert\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455729 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjhmf\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-kube-api-access-pjhmf\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455765 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-data\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.455832 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-certs\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.456326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.456234 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-certs\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.461137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.457281 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11df5d62-c1f6-4651-b068-b0bacee10d4a-istiod-ca-cert\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.461137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.457627 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-credential-socket\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.461137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.457910 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-socket\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.461137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.458166 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-data\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.461137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.458513 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-envoy\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.478338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.462925 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-podinfo\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.499436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.492628 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-token\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.499436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.492953 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjhmf\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-kube-api-access-pjhmf\") pod \"e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.564478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.564442 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:32.900378 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:37:32.900352 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11df5d62_c1f6_4651_b068_b0bacee10d4a.slice/crio-d54896326abc687d18f5c8673346209c21181778bc679b68ece868f5e466dd00 WatchSource:0}: Error finding container d54896326abc687d18f5c8673346209c21181778bc679b68ece868f5e466dd00: Status 404 returned error can't find the container with id d54896326abc687d18f5c8673346209c21181778bc679b68ece868f5e466dd00 Oct 31 21:37:32.900378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.900359 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr"] Oct 31 21:37:32.903240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.903205 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:37:32.903389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.903278 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:37:32.903389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.903324 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:37:32.946030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:32.946002 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" event={"ID":"11df5d62-c1f6-4651-b068-b0bacee10d4a","Type":"ContainerStarted","Data":"d54896326abc687d18f5c8673346209c21181778bc679b68ece868f5e466dd00"} Oct 31 21:37:33.953509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:33.953471 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" event={"ID":"11df5d62-c1f6-4651-b068-b0bacee10d4a","Type":"ContainerStarted","Data":"f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18"} Oct 31 21:37:34.014063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:34.014014 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podStartSLOduration=2.014000742 podStartE2EDuration="2.014000742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:34.012476751 +0000 UTC m=+1294.137583983" watchObservedRunningTime="2026-10-31 21:37:34.014000742 +0000 UTC m=+1294.139107972" Oct 31 21:37:34.564921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:34.564887 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:34.570244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:34.570217 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:34.960310 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:34.960220 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:34.961233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:34.961212 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:37:38.366674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.366639 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg"] Oct 31 21:37:38.370693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.370674 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.388478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.388454 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fc0d65\"" Oct 31 21:37:38.388879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.388856 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fc0d65-dockercfg-kc6g9\"" Oct 31 21:37:38.390169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.389428 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg"] Oct 31 21:37:38.520329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.499824 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08889b15-ea9d-4491-8320-430fd79c81cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.520329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.500115 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k59z\" (UniqueName: \"kubernetes.io/projected/08889b15-ea9d-4491-8320-430fd79c81cf-kube-api-access-8k59z\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.573375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.562804 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5"] Oct 31 21:37:38.573375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.569774 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5"] Oct 31 21:37:38.573375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.569882 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:38.602333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.602256 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k59z\" (UniqueName: \"kubernetes.io/projected/08889b15-ea9d-4491-8320-430fd79c81cf-kube-api-access-8k59z\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.602501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.602395 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08889b15-ea9d-4491-8320-430fd79c81cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.603493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.603468 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08889b15-ea9d-4491-8320-430fd79c81cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.631769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.631698 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k59z\" (UniqueName: \"kubernetes.io/projected/08889b15-ea9d-4491-8320-430fd79c81cf-kube-api-access-8k59z\") pod \"payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.684853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.684824 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:38.703655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.703626 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2336b5a-e893-4204-b9f8-d43124b8eef5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:38.703791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.703772 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5stxx\" (UniqueName: \"kubernetes.io/projected/e2336b5a-e893-4204-b9f8-d43124b8eef5-kube-api-access-5stxx\") pod \"payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:38.808437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.806758 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2336b5a-e893-4204-b9f8-d43124b8eef5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:38.808437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.806874 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5stxx\" (UniqueName: \"kubernetes.io/projected/e2336b5a-e893-4204-b9f8-d43124b8eef5-kube-api-access-5stxx\") pod \"payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:38.808437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.808174 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2336b5a-e893-4204-b9f8-d43124b8eef5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:38.840176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.840122 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5stxx\" (UniqueName: \"kubernetes.io/projected/e2336b5a-e893-4204-b9f8-d43124b8eef5-kube-api-access-5stxx\") pod \"payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:38.899636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:38.899535 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:39.005244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:39.005029 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg"] Oct 31 21:37:39.233553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:39.233521 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5"] Oct 31 21:37:39.242265 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:37:39.242228 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2336b5a_e893_4204_b9f8_d43124b8eef5.slice/crio-23b6c68324a5f5aabcd4bb3198436f7674d71b22e58d9f0282383387826197fc WatchSource:0}: Error finding container 23b6c68324a5f5aabcd4bb3198436f7674d71b22e58d9f0282383387826197fc: Status 404 returned error can't find the container with id 23b6c68324a5f5aabcd4bb3198436f7674d71b22e58d9f0282383387826197fc Oct 31 21:37:39.915492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:39.915452 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 21:37:39.915961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:39.915851 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 21:37:40.046130 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.046095 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" event={"ID":"e2336b5a-e893-4204-b9f8-d43124b8eef5","Type":"ContainerStarted","Data":"e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085"} Oct 31 21:37:40.046296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.046147 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" event={"ID":"e2336b5a-e893-4204-b9f8-d43124b8eef5","Type":"ContainerStarted","Data":"23b6c68324a5f5aabcd4bb3198436f7674d71b22e58d9f0282383387826197fc"} Oct 31 21:37:40.046427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.046409 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:40.048425 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.048388 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" event={"ID":"08889b15-ea9d-4491-8320-430fd79c81cf","Type":"ContainerStarted","Data":"369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622"} Oct 31 21:37:40.048552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.048435 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" event={"ID":"08889b15-ea9d-4491-8320-430fd79c81cf","Type":"ContainerStarted","Data":"ed208ca39b59cf68dca841e6f4cca7630d7b9b42d758d2cf2079604ff11c780f"} Oct 31 21:37:40.048552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.048481 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:40.147670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.147611 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" podStartSLOduration=2.147594421 podStartE2EDuration="2.147594421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:40.107195823 +0000 UTC m=+1300.232303054" watchObservedRunningTime="2026-10-31 21:37:40.147594421 +0000 UTC m=+1300.272701657" Oct 31 21:37:40.148500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:40.148462 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" podStartSLOduration=2.148454592 podStartE2EDuration="2.148454592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:40.146711263 +0000 UTC m=+1300.271818532" watchObservedRunningTime="2026-10-31 21:37:40.148454592 +0000 UTC m=+1300.273561823" Oct 31 21:37:49.923430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:49.923353 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z"] Oct 31 21:37:49.927701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:49.927676 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:49.953163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:49.952674 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z"] Oct 31 21:37:49.973447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:49.968327 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c40fc3-openshift-default-dockercfg-pk7j4\"" Oct 31 21:37:50.011527 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011494 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hm2\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-kube-api-access-77hm2\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011556 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82102cf-6d60-4be1-82e3-40c67d14215d-istiod-ca-cert\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011673 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-podinfo\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011694 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-socket\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011747 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-token\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011821 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-credential-socket\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011843 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-data\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011862 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-envoy\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.011893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.011887 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-certs\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.134982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.134939 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hm2\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-kube-api-access-77hm2\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135042 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82102cf-6d60-4be1-82e3-40c67d14215d-istiod-ca-cert\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135239 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135181 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-podinfo\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135239 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135217 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-socket\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135323 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-token\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135483 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-credential-socket\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135545 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-data\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135612 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-envoy\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.135772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.135664 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-certs\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.137670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.137643 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82102cf-6d60-4be1-82e3-40c67d14215d-istiod-ca-cert\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.142735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.142707 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-credential-socket\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.143148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.143128 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-socket\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.144199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.144173 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-data\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.164414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.157142 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-certs\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.169412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.169382 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hm2\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-kube-api-access-77hm2\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.170154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.170125 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-podinfo\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.171282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.171258 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-token\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.172895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.172866 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-envoy\") pod \"e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.250402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.250366 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:50.582701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.582667 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z"] Oct 31 21:37:50.583218 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:37:50.583188 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82102cf_6d60_4be1_82e3_40c67d14215d.slice/crio-debb535211aa259da83d880a2cf31d4a7d5eba18b9e05b7907ee8bea9775d859 WatchSource:0}: Error finding container debb535211aa259da83d880a2cf31d4a7d5eba18b9e05b7907ee8bea9775d859: Status 404 returned error can't find the container with id debb535211aa259da83d880a2cf31d4a7d5eba18b9e05b7907ee8bea9775d859 Oct 31 21:37:50.585893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.585850 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:37:50.585997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.585961 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:37:50.586059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.586010 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:37:50.608457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.608424 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686"] Oct 31 21:37:50.614675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.614656 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.630699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.630674 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fc0d65-serving-cert\"" Oct 31 21:37:50.633679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.633434 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686"] Oct 31 21:37:50.744373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.744332 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069ea2b3-55cc-45b7-9a52-95897f5c59cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.744577 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.744549 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsp9w\" (UniqueName: \"kubernetes.io/projected/069ea2b3-55cc-45b7-9a52-95897f5c59cc-kube-api-access-xsp9w\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.744700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.744682 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.744761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.744724 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.847156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.847052 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069ea2b3-55cc-45b7-9a52-95897f5c59cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.847367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.847200 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsp9w\" (UniqueName: \"kubernetes.io/projected/069ea2b3-55cc-45b7-9a52-95897f5c59cc-kube-api-access-xsp9w\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.847367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.847279 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.847493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.847450 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.847808 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:50.847784 4294 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-serving-cert: secret "maas-api-e2e-ipp-a-fc0d65-serving-cert" not found Oct 31 21:37:50.847924 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:37:50.847857 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls podName:069ea2b3-55cc-45b7-9a52-95897f5c59cc nodeName:}" failed. No retries permitted until 2026-10-31 21:37:51.347836979 +0000 UTC m=+1311.472944206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls") pod "maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" (UID: "069ea2b3-55cc-45b7-9a52-95897f5c59cc") : secret "maas-api-e2e-ipp-a-fc0d65-serving-cert" not found Oct 31 21:37:50.849367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.848784 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069ea2b3-55cc-45b7-9a52-95897f5c59cc-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.851467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.851441 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:50.878040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:50.878003 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsp9w\" (UniqueName: \"kubernetes.io/projected/069ea2b3-55cc-45b7-9a52-95897f5c59cc-kube-api-access-xsp9w\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:51.057916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.057888 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:37:51.058418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.058082 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:37:51.175532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.175435 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" event={"ID":"a82102cf-6d60-4be1-82e3-40c67d14215d","Type":"ContainerStarted","Data":"aa1f06fb7b0901b7673cfb3dc0c3daee585338e445609dab2338cace5fd5cc67"} Oct 31 21:37:51.175532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.175516 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" event={"ID":"a82102cf-6d60-4be1-82e3-40c67d14215d","Type":"ContainerStarted","Data":"debb535211aa259da83d880a2cf31d4a7d5eba18b9e05b7907ee8bea9775d859"} Oct 31 21:37:51.223014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.222965 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podStartSLOduration=2.222950425 podStartE2EDuration="2.222950425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:51.22089031 +0000 UTC m=+1311.345997543" watchObservedRunningTime="2026-10-31 21:37:51.222950425 +0000 UTC m=+1311.348057656" Oct 31 21:37:51.251448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.251419 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:51.257511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.257491 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:51.355548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.355514 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:51.371993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.371959 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:51.548104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.548070 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:51.928958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:51.928920 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686"] Oct 31 21:37:52.188618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:52.188534 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" event={"ID":"069ea2b3-55cc-45b7-9a52-95897f5c59cc","Type":"ContainerStarted","Data":"091c175b2003e945d90686e47c0a07b488b19243b4190274a05cf913af52ebbb"} Oct 31 21:37:52.188979 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:52.188834 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:52.189764 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:52.189745 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:37:53.196237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:53.196190 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" event={"ID":"069ea2b3-55cc-45b7-9a52-95897f5c59cc","Type":"ContainerStarted","Data":"3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595"} Oct 31 21:37:53.196767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:53.196404 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:37:53.245031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:53.244979 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" podStartSLOduration=2.917628097 podStartE2EDuration="3.244965533s" totalImagesPullingTime="327.337436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:37:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:37:53.241232532 +0000 UTC m=+1313.366339764" watchObservedRunningTime="2026-10-31 21:37:53.244965533 +0000 UTC m=+1313.370072765" Oct 31 21:37:59.207769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:37:59.207738 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:38:00.914411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:00.914371 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc"] Oct 31 21:38:00.914907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:00.914679 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" containerID="cri-o://a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006" gracePeriod=30 Oct 31 21:38:01.316910 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:38:01.316877 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:38:06.285758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.285729 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:38:06.344074 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.344039 4294 generic.go:356] "Generic (PLEG): container finished" podID="5d59ef2e-382a-44e7-82b0-615d84578987" containerID="a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006" exitCode=0 Oct 31 21:38:06.344227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.344083 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" Oct 31 21:38:06.344227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.344122 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" event={"ID":"5d59ef2e-382a-44e7-82b0-615d84578987","Type":"ContainerDied","Data":"a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006"} Oct 31 21:38:06.344227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.344181 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc" event={"ID":"5d59ef2e-382a-44e7-82b0-615d84578987","Type":"ContainerDied","Data":"afb5139f5e5f640ec0e96b6a3069daea47be664a9b1aebcfbe85682da79473bf"} Oct 31 21:38:06.344227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.344207 4294 scope.go:122] "RemoveContainer" containerID="a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006" Oct 31 21:38:06.356149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.356123 4294 scope.go:122] "RemoveContainer" containerID="a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006" Oct 31 21:38:06.356553 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:38:06.356533 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006\": container with ID starting with a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006 not found: ID does not exist" containerID="a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006" Oct 31 21:38:06.356643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.356560 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006"} err="failed to get container status \"a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006\": rpc error: code = NotFound desc = could not find container \"a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006\": container with ID starting with a1e041a654bdae5de40c21fba2b51d2f1438cf58785e20f2622ab0ef403fd006 not found: ID does not exist" Oct 31 21:38:06.448172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448143 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d59ef2e-382a-44e7-82b0-615d84578987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d59ef2e-382a-44e7-82b0-615d84578987-istiod-ca-cert\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448196 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-credential-socket\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448221 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-istio-token\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-istio-token\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448273 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-envoy\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448389 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-certs\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448465 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-kube-api-access-lgnzl\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-kube-api-access-lgnzl\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448505 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d59ef2e-382a-44e7-82b0-615d84578987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d59ef2e-382a-44e7-82b0-615d84578987-istio-podinfo\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448549 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-credential-socket" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.448699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448593 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d59ef2e-382a-44e7-82b0-615d84578987-istiod-ca-cert" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:06.448699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448633 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-data\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448696 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-socket\") pod \"5d59ef2e-382a-44e7-82b0-615d84578987\" (UID: \"5d59ef2e-382a-44e7-82b0-615d84578987\") " Oct 31 21:38:06.448805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448733 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-certs" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.448965 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.448947 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-data" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.449038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.449013 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-socket" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.449418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.449398 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d59ef2e-382a-44e7-82b0-615d84578987-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.449487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.449427 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.449487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.449442 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.449487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.449457 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.449487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.449470 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.450706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.450676 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d59ef2e-382a-44e7-82b0-615d84578987-istio-podinfo" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:38:06.450802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.450730 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-kube-api-access-lgnzl" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "kube-api-access-lgnzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:06.450879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.450863 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-envoy" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:38:06.450917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.450873 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-istio-token" pod "5d59ef2e-382a-44e7-82b0-615d84578987" (UID: "5d59ef2e-382a-44e7-82b0-615d84578987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:06.550698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.550621 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgnzl\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-kube-api-access-lgnzl\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.550698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.550651 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d59ef2e-382a-44e7-82b0-615d84578987-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.550698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.550660 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d59ef2e-382a-44e7-82b0-615d84578987-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.550698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.550668 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d59ef2e-382a-44e7-82b0-615d84578987-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:06.691857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.691824 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc"] Oct 31 21:38:06.703474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:06.703441 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-084d807f-openshift-default-84c8676869-9pvlc"] Oct 31 21:38:07.719823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.719790 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88"] Oct 31 21:38:07.720533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.720515 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" Oct 31 21:38:07.720533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.720532 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" Oct 31 21:38:07.720675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.720661 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" containerName="istio-proxy" Oct 31 21:38:07.725502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.725481 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:07.746124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.746100 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88"] Oct 31 21:38:07.754487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.754455 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-07afdd-dockercfg-cv9m2\"" Oct 31 21:38:07.754595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.754500 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-07afdd\"" Oct 31 21:38:07.823498 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.822736 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn"] Oct 31 21:38:07.834740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.834654 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:07.871328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.862066 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn"] Oct 31 21:38:07.871328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.868236 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7khh\" (UniqueName: \"kubernetes.io/projected/c572fa1e-45df-4f2c-8408-ccf829ce5af2-kube-api-access-f7khh\") pod \"payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:07.871328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.868367 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c572fa1e-45df-4f2c-8408-ccf829ce5af2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:07.980970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.980885 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwsj\" (UniqueName: \"kubernetes.io/projected/95d850db-c364-4312-8cca-08bc802fde07-kube-api-access-pgwsj\") pod \"payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:07.980970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.980925 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d850db-c364-4312-8cca-08bc802fde07-plugins-config-volume\") pod \"payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:07.981155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.981049 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7khh\" (UniqueName: \"kubernetes.io/projected/c572fa1e-45df-4f2c-8408-ccf829ce5af2-kube-api-access-f7khh\") pod \"payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:07.981155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.981116 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c572fa1e-45df-4f2c-8408-ccf829ce5af2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:07.981929 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.981904 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c572fa1e-45df-4f2c-8408-ccf829ce5af2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:07.988142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:07.988116 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d59ef2e-382a-44e7-82b0-615d84578987" path="/var/lib/kubelet/pods/5d59ef2e-382a-44e7-82b0-615d84578987/volumes" Oct 31 21:38:08.005366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.005342 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7khh\" (UniqueName: \"kubernetes.io/projected/c572fa1e-45df-4f2c-8408-ccf829ce5af2-kube-api-access-f7khh\") pod \"payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:08.039371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.039342 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:08.084150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.083196 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwsj\" (UniqueName: \"kubernetes.io/projected/95d850db-c364-4312-8cca-08bc802fde07-kube-api-access-pgwsj\") pod \"payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:08.084150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.083652 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d850db-c364-4312-8cca-08bc802fde07-plugins-config-volume\") pod \"payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:08.084644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.084603 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d850db-c364-4312-8cca-08bc802fde07-plugins-config-volume\") pod \"payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:08.109283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.109248 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwsj\" (UniqueName: \"kubernetes.io/projected/95d850db-c364-4312-8cca-08bc802fde07-kube-api-access-pgwsj\") pod \"payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:08.243072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.243043 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:08.354793 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.353888 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88"] Oct 31 21:38:08.561570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:08.561543 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn"] Oct 31 21:38:08.562525 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:38:08.562501 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d850db_c364_4312_8cca_08bc802fde07.slice/crio-e378e3494321d22e77ee8b95beb7fcebe5e854c0228b6c6f9d6b3f8927e7cb8d WatchSource:0}: Error finding container e378e3494321d22e77ee8b95beb7fcebe5e854c0228b6c6f9d6b3f8927e7cb8d: Status 404 returned error can't find the container with id e378e3494321d22e77ee8b95beb7fcebe5e854c0228b6c6f9d6b3f8927e7cb8d Oct 31 21:38:09.378375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.378328 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" event={"ID":"95d850db-c364-4312-8cca-08bc802fde07","Type":"ContainerStarted","Data":"b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c"} Oct 31 21:38:09.378809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.378391 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" event={"ID":"95d850db-c364-4312-8cca-08bc802fde07","Type":"ContainerStarted","Data":"e378e3494321d22e77ee8b95beb7fcebe5e854c0228b6c6f9d6b3f8927e7cb8d"} Oct 31 21:38:09.378809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.378413 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:09.379852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.379825 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" event={"ID":"c572fa1e-45df-4f2c-8408-ccf829ce5af2","Type":"ContainerStarted","Data":"d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e"} Oct 31 21:38:09.379989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.379866 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" event={"ID":"c572fa1e-45df-4f2c-8408-ccf829ce5af2","Type":"ContainerStarted","Data":"d5db70d154fea141332b38ca114b74095812b278a2b7e6221796c42003278aea"} Oct 31 21:38:09.379989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.379914 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:09.419813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.419763 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" podStartSLOduration=2.419750204 podStartE2EDuration="2.419750204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:09.418733373 +0000 UTC m=+1329.543840605" watchObservedRunningTime="2026-10-31 21:38:09.419750204 +0000 UTC m=+1329.544857453" Oct 31 21:38:09.473089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:09.473036 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" podStartSLOduration=2.473021616 podStartE2EDuration="2.473021616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:09.471559426 +0000 UTC m=+1329.596666682" watchObservedRunningTime="2026-10-31 21:38:09.473021616 +0000 UTC m=+1329.598128846" Oct 31 21:38:10.125179 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.124825 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh"] Oct 31 21:38:10.130276 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.130239 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.141588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.141238 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh"] Oct 31 21:38:10.152556 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.152484 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c40fc3-serving-cert\"" Oct 31 21:38:10.221341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.221266 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de237056-17be-4ed5-ad5f-03a2d637d066-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.221517 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.221487 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.221598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.221585 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbnp\" (UniqueName: \"kubernetes.io/projected/de237056-17be-4ed5-ad5f-03a2d637d066-kube-api-access-9tbnp\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.221679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.221667 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-metrics-certs\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.322601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.322565 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de237056-17be-4ed5-ad5f-03a2d637d066-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.322807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.322683 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.322807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.322745 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbnp\" (UniqueName: \"kubernetes.io/projected/de237056-17be-4ed5-ad5f-03a2d637d066-kube-api-access-9tbnp\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.322807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.322803 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-metrics-certs\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.323365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.323336 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de237056-17be-4ed5-ad5f-03a2d637d066-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.325359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.325329 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.325359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.325348 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-metrics-certs\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.347200 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.347165 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbnp\" (UniqueName: \"kubernetes.io/projected/de237056-17be-4ed5-ad5f-03a2d637d066-kube-api-access-9tbnp\") pod \"maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.504925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.503581 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:10.820032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:10.820008 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh"] Oct 31 21:38:10.822981 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:38:10.822954 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde237056_17be_4ed5_ad5f_03a2d637d066.slice/crio-4967043b124d8ff0b6213edebf581ef30cb46a66b913ac3cebb2b19657d43120 WatchSource:0}: Error finding container 4967043b124d8ff0b6213edebf581ef30cb46a66b913ac3cebb2b19657d43120: Status 404 returned error can't find the container with id 4967043b124d8ff0b6213edebf581ef30cb46a66b913ac3cebb2b19657d43120 Oct 31 21:38:11.452607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:11.452568 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" event={"ID":"de237056-17be-4ed5-ad5f-03a2d637d066","Type":"ContainerStarted","Data":"4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860"} Oct 31 21:38:11.452935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:11.452912 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" event={"ID":"de237056-17be-4ed5-ad5f-03a2d637d066","Type":"ContainerStarted","Data":"4967043b124d8ff0b6213edebf581ef30cb46a66b913ac3cebb2b19657d43120"} Oct 31 21:38:11.453002 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:11.452954 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:11.503372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:11.502563 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" podStartSLOduration=1.087029716 podStartE2EDuration="1.502543026s" totalImagesPullingTime="415.51331ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:11.500816089 +0000 UTC m=+1331.625923325" watchObservedRunningTime="2026-10-31 21:38:11.502543026 +0000 UTC m=+1331.627650259" Oct 31 21:38:12.645172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:12.645137 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5"] Oct 31 21:38:12.700382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:12.700344 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg"] Oct 31 21:38:13.939365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:13.936317 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28"] Oct 31 21:38:13.948347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:13.944484 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:13.968452 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:13.966095 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d75c94-openshift-default-dockercfg-j7zst\"" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.010122 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-envoy\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.010192 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-socket\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.010239 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-certs\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.010273 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-data\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.010328 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-token\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.010914 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-podinfo\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.011146 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-credential-socket\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.011222 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istiod-ca-cert\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.011325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.011264 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjbfd\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-kube-api-access-kjbfd\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.019914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.019890 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28"] Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114447 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-envoy\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114526 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-socket\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114576 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-certs\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114611 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-data\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114659 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-token\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114721 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-podinfo\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114863 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-credential-socket\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114912 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istiod-ca-cert\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.114935 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjbfd\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-kube-api-access-kjbfd\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.119325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.118071 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-data\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.135954 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.135918 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-socket\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.136173 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.136132 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-certs\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.137786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.137760 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-credential-socket\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.138119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.138025 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istiod-ca-cert\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.139881 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.139789 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-podinfo\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.140562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.140532 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-envoy\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.171083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.171047 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-token\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.187323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.175807 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjbfd\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-kube-api-access-kjbfd\") pod \"e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.270624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.270524 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:14.686826 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.686732 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28"] Oct 31 21:38:14.704464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.703779 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:14.704464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.703861 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:14.704464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:14.703908 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:15.521351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:15.521296 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" event={"ID":"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0","Type":"ContainerStarted","Data":"b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8"} Oct 31 21:38:15.521791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:15.521373 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" event={"ID":"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0","Type":"ContainerStarted","Data":"75181f1369464e4a0b81862f8c26741dcc679356b1df750818a37216fe091a7c"} Oct 31 21:38:15.585939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:15.585879 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" podStartSLOduration=2.585859992 podStartE2EDuration="2.585859992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:15.584280206 +0000 UTC m=+1335.709387436" watchObservedRunningTime="2026-10-31 21:38:15.585859992 +0000 UTC m=+1335.710967224" Oct 31 21:38:16.270813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.270771 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:16.276945 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.276912 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:16.353853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.353815 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj"] Oct 31 21:38:16.358277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.358252 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.373787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.373759 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 21:38:16.373969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.373801 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 21:38:16.373969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.373863 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 21:38:16.374135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.374116 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wxtz5\"" Oct 31 21:38:16.375842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.375818 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj"] Oct 31 21:38:16.474273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.474237 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.474454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.474317 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.474454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.474410 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.474588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.474482 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqpm9\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-kube-api-access-jqpm9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.474684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.474662 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aefdb23-64c9-4043-8ba2-fee8d84a0860-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.474820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.474806 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.474886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.474853 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.528944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.528726 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:16.530445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.530416 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:38:16.576569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.576526 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqpm9\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-kube-api-access-jqpm9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.576770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.576662 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aefdb23-64c9-4043-8ba2-fee8d84a0860-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.576830 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.576818 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.576885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.576874 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.577007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.576990 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.577067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.577037 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.577118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.577074 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.578796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.578739 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.581037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.580985 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aefdb23-64c9-4043-8ba2-fee8d84a0860-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.581644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.581617 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.582685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.582659 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.582807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.582695 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.604148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.604082 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqpm9\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-kube-api-access-jqpm9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.606869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.606814 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-kb2gj\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:16.672550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:16.672518 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:17.027504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:17.027473 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj"] Oct 31 21:38:17.029456 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:38:17.029426 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aefdb23_64c9_4043_8ba2_fee8d84a0860.slice/crio-df3b57d83ee28488ad67f5ee27fe54449dfb22e3eb8c7db1077a4e3edf59aff0 WatchSource:0}: Error finding container df3b57d83ee28488ad67f5ee27fe54449dfb22e3eb8c7db1077a4e3edf59aff0: Status 404 returned error can't find the container with id df3b57d83ee28488ad67f5ee27fe54449dfb22e3eb8c7db1077a4e3edf59aff0 Oct 31 21:38:17.466104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:17.466025 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:38:17.541415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:17.541378 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" event={"ID":"2aefdb23-64c9-4043-8ba2-fee8d84a0860","Type":"ContainerStarted","Data":"df3b57d83ee28488ad67f5ee27fe54449dfb22e3eb8c7db1077a4e3edf59aff0"} Oct 31 21:38:17.645574 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:17.645537 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" podUID="e2336b5a-e893-4204-b9f8-d43124b8eef5" containerName="payload-processing" containerID="cri-o://e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085" gracePeriod=25 Oct 31 21:38:17.700809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:17.700766 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" podUID="08889b15-ea9d-4491-8320-430fd79c81cf" containerName="payload-pre-processing" containerID="cri-o://369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622" gracePeriod=25 Oct 31 21:38:18.095895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.095858 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:38:18.100257 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.100233 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:38:18.216889 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.216859 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2336b5a-e893-4204-b9f8-d43124b8eef5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2336b5a-e893-4204-b9f8-d43124b8eef5-plugins-config-volume\") pod \"e2336b5a-e893-4204-b9f8-d43124b8eef5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " Oct 31 21:38:18.217054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.216945 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2336b5a-e893-4204-b9f8-d43124b8eef5-kube-api-access-5stxx\" (UniqueName: \"kubernetes.io/projected/e2336b5a-e893-4204-b9f8-d43124b8eef5-kube-api-access-5stxx\") pod \"e2336b5a-e893-4204-b9f8-d43124b8eef5\" (UID: \"e2336b5a-e893-4204-b9f8-d43124b8eef5\") " Oct 31 21:38:18.217054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.217028 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08889b15-ea9d-4491-8320-430fd79c81cf-kube-api-access-8k59z\" (UniqueName: \"kubernetes.io/projected/08889b15-ea9d-4491-8320-430fd79c81cf-kube-api-access-8k59z\") pod \"08889b15-ea9d-4491-8320-430fd79c81cf\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " Oct 31 21:38:18.217179 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.217152 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08889b15-ea9d-4491-8320-430fd79c81cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08889b15-ea9d-4491-8320-430fd79c81cf-plugins-config-volume\") pod \"08889b15-ea9d-4491-8320-430fd79c81cf\" (UID: \"08889b15-ea9d-4491-8320-430fd79c81cf\") " Oct 31 21:38:18.217826 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.217793 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2336b5a-e893-4204-b9f8-d43124b8eef5-plugins-config-volume" pod "e2336b5a-e893-4204-b9f8-d43124b8eef5" (UID: "e2336b5a-e893-4204-b9f8-d43124b8eef5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:18.218010 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.217985 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08889b15-ea9d-4491-8320-430fd79c81cf-plugins-config-volume" pod "08889b15-ea9d-4491-8320-430fd79c81cf" (UID: "08889b15-ea9d-4491-8320-430fd79c81cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:38:18.220762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.220740 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08889b15-ea9d-4491-8320-430fd79c81cf-kube-api-access-8k59z" pod "08889b15-ea9d-4491-8320-430fd79c81cf" (UID: "08889b15-ea9d-4491-8320-430fd79c81cf"). InnerVolumeSpecName "kube-api-access-8k59z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:18.222481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.222452 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2336b5a-e893-4204-b9f8-d43124b8eef5-kube-api-access-5stxx" pod "e2336b5a-e893-4204-b9f8-d43124b8eef5" (UID: "e2336b5a-e893-4204-b9f8-d43124b8eef5"). InnerVolumeSpecName "kube-api-access-5stxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:38:18.319616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.319507 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2336b5a-e893-4204-b9f8-d43124b8eef5-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:18.319616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.319545 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5stxx\" (UniqueName: \"kubernetes.io/projected/e2336b5a-e893-4204-b9f8-d43124b8eef5-kube-api-access-5stxx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:18.319616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.319561 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k59z\" (UniqueName: \"kubernetes.io/projected/08889b15-ea9d-4491-8320-430fd79c81cf-kube-api-access-8k59z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:18.319616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.319575 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08889b15-ea9d-4491-8320-430fd79c81cf-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:38:18.550028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.549990 4294 generic.go:356] "Generic (PLEG): container finished" podID="e2336b5a-e893-4204-b9f8-d43124b8eef5" containerID="e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085" exitCode=0 Oct 31 21:38:18.550532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.550040 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" Oct 31 21:38:18.550532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.550069 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" event={"ID":"e2336b5a-e893-4204-b9f8-d43124b8eef5","Type":"ContainerDied","Data":"e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085"} Oct 31 21:38:18.550532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.550138 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5" event={"ID":"e2336b5a-e893-4204-b9f8-d43124b8eef5","Type":"ContainerDied","Data":"23b6c68324a5f5aabcd4bb3198436f7674d71b22e58d9f0282383387826197fc"} Oct 31 21:38:18.550532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.550191 4294 scope.go:122] "RemoveContainer" containerID="e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085" Oct 31 21:38:18.551615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.551563 4294 generic.go:356] "Generic (PLEG): container finished" podID="08889b15-ea9d-4491-8320-430fd79c81cf" containerID="369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622" exitCode=0 Oct 31 21:38:18.551615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.551612 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" Oct 31 21:38:18.551730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.551647 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" event={"ID":"08889b15-ea9d-4491-8320-430fd79c81cf","Type":"ContainerDied","Data":"369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622"} Oct 31 21:38:18.551730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.551699 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg" event={"ID":"08889b15-ea9d-4491-8320-430fd79c81cf","Type":"ContainerDied","Data":"ed208ca39b59cf68dca841e6f4cca7630d7b9b42d758d2cf2079604ff11c780f"} Oct 31 21:38:18.567609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.567590 4294 scope.go:122] "RemoveContainer" containerID="e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085" Oct 31 21:38:18.567922 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:38:18.567901 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085\": container with ID starting with e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085 not found: ID does not exist" containerID="e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085" Oct 31 21:38:18.568009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.567931 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085"} err="failed to get container status \"e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085\": rpc error: code = NotFound desc = could not find container \"e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085\": container with ID starting with e4be89a30846d80ddb729ced78bcad231e808ab6e29fd2a4d0b74416a734f085 not found: ID does not exist" Oct 31 21:38:18.568009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.567955 4294 scope.go:122] "RemoveContainer" containerID="369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622" Oct 31 21:38:18.592840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.592813 4294 scope.go:122] "RemoveContainer" containerID="369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622" Oct 31 21:38:18.593195 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:38:18.593167 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622\": container with ID starting with 369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622 not found: ID does not exist" containerID="369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622" Oct 31 21:38:18.593507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.593202 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622"} err="failed to get container status \"369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622\": rpc error: code = NotFound desc = could not find container \"369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622\": container with ID starting with 369b6fbe9b88c5312806ffbb96b4b404bf0f4c9464bda0dfef377bcb4bde3622 not found: ID does not exist" Oct 31 21:38:18.629351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.629122 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg"] Oct 31 21:38:18.637487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.637443 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc0d65-6f75b86795-mf9jg"] Oct 31 21:38:18.685091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.685059 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5"] Oct 31 21:38:18.688691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:18.688667 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc0d65-5657d46c4f-ql4x5"] Oct 31 21:38:19.992945 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:19.992897 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08889b15-ea9d-4491-8320-430fd79c81cf" path="/var/lib/kubelet/pods/08889b15-ea9d-4491-8320-430fd79c81cf/volumes" Oct 31 21:38:20.294947 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:19.993687 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2336b5a-e893-4204-b9f8-d43124b8eef5" path="/var/lib/kubelet/pods/e2336b5a-e893-4204-b9f8-d43124b8eef5/volumes" Oct 31 21:38:20.384023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.383986 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:20.384176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.384058 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:20.419343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.414341 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:38:20.419343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.414833 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:38:20.471676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.471647 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": read tcp 10.134.0.2:39948->10.134.0.85:15021: read: connection reset by peer" start-of-body= Oct 31 21:38:20.471797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.471729 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": read tcp 10.134.0.2:39948->10.134.0.85:15021: read: connection reset by peer" Oct 31 21:38:20.590081 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.589837 4294 generic.go:356] "Generic (PLEG): container finished" podID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerID="f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18" exitCode=0 Oct 31 21:38:20.590081 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.589899 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" event={"ID":"11df5d62-c1f6-4651-b068-b0bacee10d4a","Type":"ContainerDied","Data":"f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18"} Oct 31 21:38:20.590857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.590832 4294 scope.go:122] "RemoveContainer" containerID="f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18" Oct 31 21:38:20.592857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.592819 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:20.593092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.593066 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:20.593206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.593123 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:20.600695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.600664 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" event={"ID":"2aefdb23-64c9-4043-8ba2-fee8d84a0860","Type":"ContainerStarted","Data":"c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645"} Oct 31 21:38:20.600801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:20.600726 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:21.632787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:21.631761 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" event={"ID":"11df5d62-c1f6-4651-b068-b0bacee10d4a","Type":"ContainerStarted","Data":"6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e"} Oct 31 21:38:21.636320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:21.636267 4294 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-kb2gj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:38:21.636485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:21.636348 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" podUID="2aefdb23-64c9-4043-8ba2-fee8d84a0860" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:38:21.708391 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:21.708331 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" podStartSLOduration=2.357441728 podStartE2EDuration="5.708292404s" totalImagesPullingTime="3.350850676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:38:20.680495465 +0000 UTC m=+1340.805602697" watchObservedRunningTime="2026-10-31 21:38:21.708292404 +0000 UTC m=+1341.833399635" Oct 31 21:38:22.565352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:22.565319 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:38:22.565549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:22.565524 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 21:38:22.565633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:22.565584 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 21:38:23.564777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:23.564739 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 21:38:23.565157 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:23.564799 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 21:38:24.565442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:24.565409 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 21:38:24.565822 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:24.565467 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 21:38:24.633655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:24.633626 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:38:25.565092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:25.565058 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 21:38:25.565268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:25.565111 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 21:38:26.565540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:26.565504 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 21:38:26.565940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:26.565561 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 21:38:27.569437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:27.569407 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:38:27.569906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:27.569883 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:38:27.570672 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:27.570652 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:38:31.329327 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:38:31.328913 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:38:38.231468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:38.231420 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:38:38.231956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:38.231487 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:38:39.876370 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:39.876244 4294 generic.go:356] "Generic (PLEG): container finished" podID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerID="aa1f06fb7b0901b7673cfb3dc0c3daee585338e445609dab2338cace5fd5cc67" exitCode=0 Oct 31 21:38:39.876786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:39.876359 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" event={"ID":"a82102cf-6d60-4be1-82e3-40c67d14215d","Type":"ContainerDied","Data":"aa1f06fb7b0901b7673cfb3dc0c3daee585338e445609dab2338cace5fd5cc67"} Oct 31 21:38:39.876849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:39.876834 4294 scope.go:122] "RemoveContainer" containerID="aa1f06fb7b0901b7673cfb3dc0c3daee585338e445609dab2338cace5fd5cc67" Oct 31 21:38:39.878545 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:39.878520 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:39.878615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:39.878586 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:39.878654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:39.878618 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:38:40.904216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:40.904180 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" event={"ID":"a82102cf-6d60-4be1-82e3-40c67d14215d","Type":"ContainerStarted","Data":"1a0c1329f9eec6ebda50328f755eee3f02f9b85f461a5a57c67ef4851e9b8cd3"} Oct 31 21:38:41.251477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:41.251437 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:38:41.252926 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:41.252891 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 21:38:41.253061 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:41.252952 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 21:38:42.254326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:42.250862 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 21:38:42.254326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:42.250922 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 21:38:43.252216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:43.250712 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 21:38:43.252216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:43.250764 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 21:38:44.250922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:44.250891 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 21:38:44.251371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:44.250944 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 21:38:45.250808 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:45.250772 4294 patch_prober.go:29] interesting pod/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 21:38:45.250977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:45.250827 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 21:38:46.255468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:46.255439 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:38:46.256058 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:46.255885 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:38:46.256590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:38:46.256570 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:39:01.469004 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:39:01.468968 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:39:07.309226 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:07.309192 4294 generic.go:356] "Generic (PLEG): container finished" podID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerID="b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8" exitCode=0 Oct 31 21:39:07.309691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:07.309259 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" event={"ID":"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0","Type":"ContainerDied","Data":"b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8"} Oct 31 21:39:07.309739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:07.309725 4294 scope.go:122] "RemoveContainer" containerID="b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8" Oct 31 21:39:07.311382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:07.311354 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:39:07.311447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:07.311421 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:39:07.311497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:07.311454 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:39:08.317887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:08.317848 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" event={"ID":"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0","Type":"ContainerStarted","Data":"3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a"} Oct 31 21:39:09.271378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:09.271271 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:39:09.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:09.286656 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 21:39:09.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:09.286729 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 21:39:09.700604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:09.700503 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5"] Oct 31 21:39:09.701065 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:09.700773 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" containerID="cri-o://cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa" gracePeriod=30 Oct 31 21:39:10.287790 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:10.287748 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 21:39:10.296463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:10.296423 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 21:39:11.284368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:11.282680 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 21:39:11.284368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:11.282757 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 21:39:12.288326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:12.275643 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 21:39:12.288326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:12.275715 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 21:39:13.271192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:13.271158 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 31 21:39:13.271427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:13.271224 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 31 21:39:14.275364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:14.275330 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:39:14.275865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:14.275839 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:39:14.276678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:14.276658 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:39:15.074204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.074169 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:39:15.126538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.126380 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-data\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.126538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.126495 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-certs\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.126835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.126599 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-envoy\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.126835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.126723 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-token\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-token\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.126835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.126767 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-credential-socket\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.126988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.126875 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e81bf27-72f7-4a89-8908-49f0c75c0306-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81bf27-72f7-4a89-8908-49f0c75c0306-istiod-ca-cert\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.126988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.126902 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-podinfo\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.127091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.127065 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-kube-api-access-4vmxb\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-kube-api-access-4vmxb\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.127414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.127243 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-socket\") pod \"8e81bf27-72f7-4a89-8908-49f0c75c0306\" (UID: \"8e81bf27-72f7-4a89-8908-49f0c75c0306\") " Oct 31 21:39:15.128410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.128102 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e81bf27-72f7-4a89-8908-49f0c75c0306-istiod-ca-cert" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:15.128632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.128569 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-data" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:15.128906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.128621 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-certs" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:15.129467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.128987 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-socket" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:15.129467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.129038 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-credential-socket" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:15.131510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.131482 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-kube-api-access-4vmxb" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "kube-api-access-4vmxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:15.131915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.131872 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-envoy" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:15.132186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.132157 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-token" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:15.132405 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.132384 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-podinfo" pod "8e81bf27-72f7-4a89-8908-49f0c75c0306" (UID: "8e81bf27-72f7-4a89-8908-49f0c75c0306"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:39:15.230080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230050 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e81bf27-72f7-4a89-8908-49f0c75c0306-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230077 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230089 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vmxb\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-kube-api-access-4vmxb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230098 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230106 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230116 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230123 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230131 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e81bf27-72f7-4a89-8908-49f0c75c0306-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.230279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.230139 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e81bf27-72f7-4a89-8908-49f0c75c0306-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:15.382891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.382805 4294 generic.go:356] "Generic (PLEG): container finished" podID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerID="cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa" exitCode=0 Oct 31 21:39:15.382891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.382855 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" Oct 31 21:39:15.383433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.382889 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" event={"ID":"8e81bf27-72f7-4a89-8908-49f0c75c0306","Type":"ContainerDied","Data":"cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa"} Oct 31 21:39:15.383433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.382948 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5" event={"ID":"8e81bf27-72f7-4a89-8908-49f0c75c0306","Type":"ContainerDied","Data":"0baf2ca7a7f0a33c8b88fcc2470217b82b96c8e4211edee55f81e288ef5d8da8"} Oct 31 21:39:15.383433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.382977 4294 scope.go:122] "RemoveContainer" containerID="cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa" Oct 31 21:39:15.393961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.393943 4294 scope.go:122] "RemoveContainer" containerID="cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa" Oct 31 21:39:15.394235 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:39:15.394215 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa\": container with ID starting with cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa not found: ID does not exist" containerID="cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa" Oct 31 21:39:15.394294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.394247 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa"} err="failed to get container status \"cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa\": rpc error: code = NotFound desc = could not find container \"cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa\": container with ID starting with cb2a517ffd6dc85d3c170ad3cdd744de40e3657329c03c8f359e29052ec45aaa not found: ID does not exist" Oct 31 21:39:15.436698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.436668 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5"] Oct 31 21:39:15.441511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.441483 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-tqtn5"] Oct 31 21:39:15.985411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:15.985381 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" path="/var/lib/kubelet/pods/8e81bf27-72f7-4a89-8908-49f0c75c0306/volumes" Oct 31 21:39:20.605602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:20.605568 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww"] Oct 31 21:39:20.606138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:20.605793 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" containerID="cri-o://39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff" gracePeriod=30 Oct 31 21:39:25.878660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:25.878601 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:39:26.032845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.032805 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-certs\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.032877 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-data\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.032924 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-podinfo\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.032975 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-socket\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033025 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-kube-api-access-v6kzx\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-kube-api-access-v6kzx\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033116 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-token\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-token\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033122 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-certs" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:26.033156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033139 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-data" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:26.033456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033211 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-envoy\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033243 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-credential-socket\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033362 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istiod-ca-cert\") pod \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\" (UID: \"10e16965-03a7-4bc3-a07d-0ed3761cb62d\") " Oct 31 21:39:26.033456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033360 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-socket" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:26.033651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033563 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-credential-socket" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:26.033786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.033762 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istiod-ca-cert" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:39:26.034144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.034124 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.034216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.034150 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.034216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.034166 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.034216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.034180 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.034216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.034193 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.035439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.035418 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-token" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:26.035439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.035424 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-envoy" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:39:26.035578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.035443 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-kube-api-access-v6kzx" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "kube-api-access-v6kzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:39:26.035578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.035418 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-podinfo" pod "10e16965-03a7-4bc3-a07d-0ed3761cb62d" (UID: "10e16965-03a7-4bc3-a07d-0ed3761cb62d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:39:26.135771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.135694 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.135771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.135721 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.135771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.135731 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10e16965-03a7-4bc3-a07d-0ed3761cb62d-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.135771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.135740 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6kzx\" (UniqueName: \"kubernetes.io/projected/10e16965-03a7-4bc3-a07d-0ed3761cb62d-kube-api-access-v6kzx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:39:26.471681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.471593 4294 generic.go:356] "Generic (PLEG): container finished" podID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerID="39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff" exitCode=0 Oct 31 21:39:26.471681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.471636 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" Oct 31 21:39:26.471915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.471676 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" event={"ID":"10e16965-03a7-4bc3-a07d-0ed3761cb62d","Type":"ContainerDied","Data":"39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff"} Oct 31 21:39:26.471915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.471742 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww" event={"ID":"10e16965-03a7-4bc3-a07d-0ed3761cb62d","Type":"ContainerDied","Data":"751ebf31bb8d7110cb8f80f90af3fda26e3640838a2fc543ce8e72b82a1523b6"} Oct 31 21:39:26.471915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.471773 4294 scope.go:122] "RemoveContainer" containerID="39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff" Oct 31 21:39:26.482824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.482807 4294 scope.go:122] "RemoveContainer" containerID="39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff" Oct 31 21:39:26.483102 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:39:26.483083 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff\": container with ID starting with 39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff not found: ID does not exist" containerID="39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff" Oct 31 21:39:26.483183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.483124 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff"} err="failed to get container status \"39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff\": rpc error: code = NotFound desc = could not find container \"39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff\": container with ID starting with 39c7a077dac072744169d7b33032228004df986330a03f63441c7698d28f7aff not found: ID does not exist" Oct 31 21:39:26.525257 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.525232 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww"] Oct 31 21:39:26.542120 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:26.542085 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-hf9ww"] Oct 31 21:39:27.985045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:39:27.985008 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" path="/var/lib/kubelet/pods/10e16965-03a7-4bc3-a07d-0ed3761cb62d/volumes" Oct 31 21:39:31.310923 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:39:31.310888 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:40:01.318764 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:40:01.318721 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:40:09.613511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.613478 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt"] Oct 31 21:40:09.614102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614085 4294 cpu_manager.go:414] "containerMap: removing container" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" Oct 31 21:40:09.614158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614106 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" Oct 31 21:40:09.614158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614132 4294 cpu_manager.go:414] "containerMap: removing container" podUID="08889b15-ea9d-4491-8320-430fd79c81cf" containerName="payload-pre-processing" Oct 31 21:40:09.614158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614137 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08889b15-ea9d-4491-8320-430fd79c81cf" containerName="payload-pre-processing" Oct 31 21:40:09.614158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614147 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" Oct 31 21:40:09.614158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614152 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" Oct 31 21:40:09.614349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614162 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e2336b5a-e893-4204-b9f8-d43124b8eef5" containerName="payload-processing" Oct 31 21:40:09.614349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614168 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2336b5a-e893-4204-b9f8-d43124b8eef5" containerName="payload-processing" Oct 31 21:40:09.614349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614267 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="08889b15-ea9d-4491-8320-430fd79c81cf" containerName="payload-pre-processing" Oct 31 21:40:09.614349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614275 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e81bf27-72f7-4a89-8908-49f0c75c0306" containerName="istio-proxy" Oct 31 21:40:09.614349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614283 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2336b5a-e893-4204-b9f8-d43124b8eef5" containerName="payload-processing" Oct 31 21:40:09.614349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.614294 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e16965-03a7-4bc3-a07d-0ed3761cb62d" containerName="istio-proxy" Oct 31 21:40:09.618815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.618797 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.642346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.640324 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt"] Oct 31 21:40:09.643567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.643545 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:40:09.644180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.644155 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:40:09.734629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.734592 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jb2v\" (UniqueName: \"kubernetes.io/projected/cc8f38e2-abab-4b78-8298-f4318afd5eba-kube-api-access-9jb2v\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.734819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.734654 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.734909 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.734878 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.835884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.835844 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jb2v\" (UniqueName: \"kubernetes.io/projected/cc8f38e2-abab-4b78-8298-f4318afd5eba-kube-api-access-9jb2v\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.836054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.835921 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.836054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.836038 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.836707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.836685 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.838252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.838230 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.864323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.864250 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jb2v\" (UniqueName: \"kubernetes.io/projected/cc8f38e2-abab-4b78-8298-f4318afd5eba-kube-api-access-9jb2v\") pod \"maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:09.932143 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:09.932112 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:10.238178 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:10.238147 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt"] Oct 31 21:40:10.238914 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:40:10.238891 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8f38e2_abab_4b78_8298_f4318afd5eba.slice/crio-0bccb5343b6d1d49a7de39a02235dc3708442cc9b712dfe8cf8a6f921756899b WatchSource:0}: Error finding container 0bccb5343b6d1d49a7de39a02235dc3708442cc9b712dfe8cf8a6f921756899b: Status 404 returned error can't find the container with id 0bccb5343b6d1d49a7de39a02235dc3708442cc9b712dfe8cf8a6f921756899b Oct 31 21:40:10.833774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:10.833686 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" event={"ID":"cc8f38e2-abab-4b78-8298-f4318afd5eba","Type":"ContainerStarted","Data":"98fe6aaaf681729ca5c87b21a3d4f7c19d107b72eda264db28273e3e676ab3ce"} Oct 31 21:40:10.833774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:10.833749 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" event={"ID":"cc8f38e2-abab-4b78-8298-f4318afd5eba","Type":"ContainerStarted","Data":"0bccb5343b6d1d49a7de39a02235dc3708442cc9b712dfe8cf8a6f921756899b"} Oct 31 21:40:10.874907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:10.874860 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" podStartSLOduration=1.874843878 podStartE2EDuration="1.874843878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:40:10.873738486 +0000 UTC m=+1450.998845719" watchObservedRunningTime="2026-10-31 21:40:10.874843878 +0000 UTC m=+1450.999951140" Oct 31 21:40:11.843430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:11.843393 4294 generic.go:356] "Generic (PLEG): container finished" podID="cc8f38e2-abab-4b78-8298-f4318afd5eba" containerID="98fe6aaaf681729ca5c87b21a3d4f7c19d107b72eda264db28273e3e676ab3ce" exitCode=0 Oct 31 21:40:11.843897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:11.843490 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" event={"ID":"cc8f38e2-abab-4b78-8298-f4318afd5eba","Type":"ContainerDied","Data":"98fe6aaaf681729ca5c87b21a3d4f7c19d107b72eda264db28273e3e676ab3ce"} Oct 31 21:40:13.013374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.013349 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:13.082637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.082606 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-openshift-ca\") pod \"cc8f38e2-abab-4b78-8298-f4318afd5eba\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " Oct 31 21:40:13.082817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.082648 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-xks-ca\") pod \"cc8f38e2-abab-4b78-8298-f4318afd5eba\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " Oct 31 21:40:13.082817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.082780 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8f38e2-abab-4b78-8298-f4318afd5eba-kube-api-access-9jb2v\" (UniqueName: \"kubernetes.io/projected/cc8f38e2-abab-4b78-8298-f4318afd5eba-kube-api-access-9jb2v\") pod \"cc8f38e2-abab-4b78-8298-f4318afd5eba\" (UID: \"cc8f38e2-abab-4b78-8298-f4318afd5eba\") " Oct 31 21:40:13.083024 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.083003 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-openshift-ca" pod "cc8f38e2-abab-4b78-8298-f4318afd5eba" (UID: "cc8f38e2-abab-4b78-8298-f4318afd5eba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:40:13.083491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.083470 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:13.084875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.084851 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-xks-ca" pod "cc8f38e2-abab-4b78-8298-f4318afd5eba" (UID: "cc8f38e2-abab-4b78-8298-f4318afd5eba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:40:13.085228 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.085203 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8f38e2-abab-4b78-8298-f4318afd5eba-kube-api-access-9jb2v" pod "cc8f38e2-abab-4b78-8298-f4318afd5eba" (UID: "cc8f38e2-abab-4b78-8298-f4318afd5eba"). InnerVolumeSpecName "kube-api-access-9jb2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:40:13.184644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.184570 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f38e2-abab-4b78-8298-f4318afd5eba-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:13.184644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.184595 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jb2v\" (UniqueName: \"kubernetes.io/projected/cc8f38e2-abab-4b78-8298-f4318afd5eba-kube-api-access-9jb2v\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:13.857432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.857404 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" Oct 31 21:40:13.857601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.857446 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt" event={"ID":"cc8f38e2-abab-4b78-8298-f4318afd5eba","Type":"ContainerDied","Data":"0bccb5343b6d1d49a7de39a02235dc3708442cc9b712dfe8cf8a6f921756899b"} Oct 31 21:40:13.857601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:13.857520 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bccb5343b6d1d49a7de39a02235dc3708442cc9b712dfe8cf8a6f921756899b" Oct 31 21:40:20.154541 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:20.154507 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn"] Oct 31 21:40:20.188656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:20.188624 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88"] Oct 31 21:40:25.155810 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.155766 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" podUID="95d850db-c364-4312-8cca-08bc802fde07" containerName="payload-processing" containerID="cri-o://b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c" gracePeriod=25 Oct 31 21:40:25.189857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.189818 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" podUID="c572fa1e-45df-4f2c-8408-ccf829ce5af2" containerName="payload-pre-processing" containerID="cri-o://d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e" gracePeriod=25 Oct 31 21:40:25.479036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.479005 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:40:25.527738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.527708 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:40:25.548980 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.548938 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d850db-c364-4312-8cca-08bc802fde07-kube-api-access-pgwsj\" (UniqueName: \"kubernetes.io/projected/95d850db-c364-4312-8cca-08bc802fde07-kube-api-access-pgwsj\") pod \"95d850db-c364-4312-8cca-08bc802fde07\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " Oct 31 21:40:25.549147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.549031 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d850db-c364-4312-8cca-08bc802fde07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d850db-c364-4312-8cca-08bc802fde07-plugins-config-volume\") pod \"95d850db-c364-4312-8cca-08bc802fde07\" (UID: \"95d850db-c364-4312-8cca-08bc802fde07\") " Oct 31 21:40:25.549435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.549407 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d850db-c364-4312-8cca-08bc802fde07-plugins-config-volume" pod "95d850db-c364-4312-8cca-08bc802fde07" (UID: "95d850db-c364-4312-8cca-08bc802fde07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:40:25.549836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.549813 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d850db-c364-4312-8cca-08bc802fde07-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:25.551253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.551228 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d850db-c364-4312-8cca-08bc802fde07-kube-api-access-pgwsj" pod "95d850db-c364-4312-8cca-08bc802fde07" (UID: "95d850db-c364-4312-8cca-08bc802fde07"). InnerVolumeSpecName "kube-api-access-pgwsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:40:25.650237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.650200 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c572fa1e-45df-4f2c-8408-ccf829ce5af2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c572fa1e-45df-4f2c-8408-ccf829ce5af2-plugins-config-volume\") pod \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " Oct 31 21:40:25.650446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.650343 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c572fa1e-45df-4f2c-8408-ccf829ce5af2-kube-api-access-f7khh\" (UniqueName: \"kubernetes.io/projected/c572fa1e-45df-4f2c-8408-ccf829ce5af2-kube-api-access-f7khh\") pod \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\" (UID: \"c572fa1e-45df-4f2c-8408-ccf829ce5af2\") " Oct 31 21:40:25.650830 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.650800 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c572fa1e-45df-4f2c-8408-ccf829ce5af2-plugins-config-volume" pod "c572fa1e-45df-4f2c-8408-ccf829ce5af2" (UID: "c572fa1e-45df-4f2c-8408-ccf829ce5af2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:40:25.650937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.650908 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c572fa1e-45df-4f2c-8408-ccf829ce5af2-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:25.650937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.650930 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgwsj\" (UniqueName: \"kubernetes.io/projected/95d850db-c364-4312-8cca-08bc802fde07-kube-api-access-pgwsj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:25.652619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.652595 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c572fa1e-45df-4f2c-8408-ccf829ce5af2-kube-api-access-f7khh" pod "c572fa1e-45df-4f2c-8408-ccf829ce5af2" (UID: "c572fa1e-45df-4f2c-8408-ccf829ce5af2"). InnerVolumeSpecName "kube-api-access-f7khh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:40:25.751849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.751819 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7khh\" (UniqueName: \"kubernetes.io/projected/c572fa1e-45df-4f2c-8408-ccf829ce5af2-kube-api-access-f7khh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:25.953969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.953937 4294 generic.go:356] "Generic (PLEG): container finished" podID="95d850db-c364-4312-8cca-08bc802fde07" containerID="b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c" exitCode=0 Oct 31 21:40:25.954154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.954018 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" Oct 31 21:40:25.954154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.954029 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" event={"ID":"95d850db-c364-4312-8cca-08bc802fde07","Type":"ContainerDied","Data":"b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c"} Oct 31 21:40:25.954154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.954087 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn" event={"ID":"95d850db-c364-4312-8cca-08bc802fde07","Type":"ContainerDied","Data":"e378e3494321d22e77ee8b95beb7fcebe5e854c0228b6c6f9d6b3f8927e7cb8d"} Oct 31 21:40:25.954154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.954118 4294 scope.go:122] "RemoveContainer" containerID="b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c" Oct 31 21:40:25.955586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.955558 4294 generic.go:356] "Generic (PLEG): container finished" podID="c572fa1e-45df-4f2c-8408-ccf829ce5af2" containerID="d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e" exitCode=0 Oct 31 21:40:25.955706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.955601 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" Oct 31 21:40:25.955706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.955593 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" event={"ID":"c572fa1e-45df-4f2c-8408-ccf829ce5af2","Type":"ContainerDied","Data":"d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e"} Oct 31 21:40:25.955706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.955656 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88" event={"ID":"c572fa1e-45df-4f2c-8408-ccf829ce5af2","Type":"ContainerDied","Data":"d5db70d154fea141332b38ca114b74095812b278a2b7e6221796c42003278aea"} Oct 31 21:40:25.964445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.964427 4294 scope.go:122] "RemoveContainer" containerID="b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c" Oct 31 21:40:25.964752 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:40:25.964698 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c\": container with ID starting with b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c not found: ID does not exist" containerID="b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c" Oct 31 21:40:25.964752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.964724 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c"} err="failed to get container status \"b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c\": rpc error: code = NotFound desc = could not find container \"b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c\": container with ID starting with b046f35933ce9298e4d71ee17488ab881837253c5d510d458e62c8136bc77d2c not found: ID does not exist" Oct 31 21:40:25.964891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.964773 4294 scope.go:122] "RemoveContainer" containerID="d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e" Oct 31 21:40:25.975445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.975422 4294 scope.go:122] "RemoveContainer" containerID="d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e" Oct 31 21:40:25.980126 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:40:25.980099 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e\": container with ID starting with d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e not found: ID does not exist" containerID="d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e" Oct 31 21:40:25.980237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:25.980136 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e"} err="failed to get container status \"d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e\": rpc error: code = NotFound desc = could not find container \"d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e\": container with ID starting with d96a7abb48b0a5586d9ac41365befe6510488bf864ebd37886fdb845f01e938e not found: ID does not exist" Oct 31 21:40:26.008753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:26.008681 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88"] Oct 31 21:40:26.021361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:26.021336 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-07afdd-696875d767-mll88"] Oct 31 21:40:26.055661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:26.055630 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn"] Oct 31 21:40:26.059888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:26.059865 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-07afdd-7f68948dcf-kspdn"] Oct 31 21:40:27.984945 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:27.984911 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d850db-c364-4312-8cca-08bc802fde07" path="/var/lib/kubelet/pods/95d850db-c364-4312-8cca-08bc802fde07/volumes" Oct 31 21:40:27.985328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:27.985286 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c572fa1e-45df-4f2c-8408-ccf829ce5af2" path="/var/lib/kubelet/pods/c572fa1e-45df-4f2c-8408-ccf829ce5af2/volumes" Oct 31 21:40:31.310709 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:40:31.310672 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:40:35.051152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051119 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg"] Oct 31 21:40:35.051712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051698 4294 cpu_manager.go:414] "containerMap: removing container" podUID="95d850db-c364-4312-8cca-08bc802fde07" containerName="payload-processing" Oct 31 21:40:35.051755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051712 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d850db-c364-4312-8cca-08bc802fde07" containerName="payload-processing" Oct 31 21:40:35.051755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051732 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c572fa1e-45df-4f2c-8408-ccf829ce5af2" containerName="payload-pre-processing" Oct 31 21:40:35.051755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051736 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c572fa1e-45df-4f2c-8408-ccf829ce5af2" containerName="payload-pre-processing" Oct 31 21:40:35.051861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051762 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cc8f38e2-abab-4b78-8298-f4318afd5eba" containerName="revoke-keys" Oct 31 21:40:35.051861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051766 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8f38e2-abab-4b78-8298-f4318afd5eba" containerName="revoke-keys" Oct 31 21:40:35.051861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051856 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8f38e2-abab-4b78-8298-f4318afd5eba" containerName="revoke-keys" Oct 31 21:40:35.051998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051867 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d850db-c364-4312-8cca-08bc802fde07" containerName="payload-processing" Oct 31 21:40:35.051998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.051874 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c572fa1e-45df-4f2c-8408-ccf829ce5af2" containerName="payload-pre-processing" Oct 31 21:40:35.056867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.056851 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.074895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.074863 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg"] Oct 31 21:40:35.081727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.081700 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:40:35.081869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.081836 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:40:35.095028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.095001 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.095192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.095146 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwddk\" (UniqueName: \"kubernetes.io/projected/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-kube-api-access-vwddk\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.095528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.095477 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.197422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.197385 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.197577 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.197493 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.197577 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.197517 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwddk\" (UniqueName: \"kubernetes.io/projected/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-kube-api-access-vwddk\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.198190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.198160 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.199756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.199736 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.220242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.220217 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwddk\" (UniqueName: \"kubernetes.io/projected/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-kube-api-access-vwddk\") pod \"maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.370526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.370453 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:35.680367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:35.675316 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg"] Oct 31 21:40:36.024151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:36.024123 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" event={"ID":"27fdd9bc-1d18-4493-b7a6-f49473c74ef2","Type":"ContainerStarted","Data":"01967b1f7621b2ec3ecff9ef88336e540f75eb17f35526b825cbf12aef98f278"} Oct 31 21:40:36.024275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:36.024178 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" event={"ID":"27fdd9bc-1d18-4493-b7a6-f49473c74ef2","Type":"ContainerStarted","Data":"e628a4ebecf014b321ce63d2e42081a841dd38f2e546f72b5e85f9fd1b2fe031"} Oct 31 21:40:36.065829 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:36.065784 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" podStartSLOduration=1.065771169 podStartE2EDuration="1.065771169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:40:36.064038265 +0000 UTC m=+1476.189145498" watchObservedRunningTime="2026-10-31 21:40:36.065771169 +0000 UTC m=+1476.190878433" Oct 31 21:40:36.628937 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:40:36.628900 4294 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fdd9bc_1d18_4493_b7a6_f49473c74ef2.slice/crio-01967b1f7621b2ec3ecff9ef88336e540f75eb17f35526b825cbf12aef98f278.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fdd9bc_1d18_4493_b7a6_f49473c74ef2.slice/crio-conmon-01967b1f7621b2ec3ecff9ef88336e540f75eb17f35526b825cbf12aef98f278.scope\": RecentStats: unable to find data in memory cache]" Oct 31 21:40:37.031932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:37.031901 4294 generic.go:356] "Generic (PLEG): container finished" podID="27fdd9bc-1d18-4493-b7a6-f49473c74ef2" containerID="01967b1f7621b2ec3ecff9ef88336e540f75eb17f35526b825cbf12aef98f278" exitCode=0 Oct 31 21:40:37.032094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:37.031985 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" event={"ID":"27fdd9bc-1d18-4493-b7a6-f49473c74ef2","Type":"ContainerDied","Data":"01967b1f7621b2ec3ecff9ef88336e540f75eb17f35526b825cbf12aef98f278"} Oct 31 21:40:38.195335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.195284 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:38.240553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.240520 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-xks-ca\") pod \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " Oct 31 21:40:38.240736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.240571 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-kube-api-access-vwddk\" (UniqueName: \"kubernetes.io/projected/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-kube-api-access-vwddk\") pod \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " Oct 31 21:40:38.240736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.240707 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-openshift-ca\") pod \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\" (UID: \"27fdd9bc-1d18-4493-b7a6-f49473c74ef2\") " Oct 31 21:40:38.241606 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.241344 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-openshift-ca" pod "27fdd9bc-1d18-4493-b7a6-f49473c74ef2" (UID: "27fdd9bc-1d18-4493-b7a6-f49473c74ef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:40:38.247791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.243747 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-xks-ca" pod "27fdd9bc-1d18-4493-b7a6-f49473c74ef2" (UID: "27fdd9bc-1d18-4493-b7a6-f49473c74ef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:40:38.247791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.244468 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-kube-api-access-vwddk" pod "27fdd9bc-1d18-4493-b7a6-f49473c74ef2" (UID: "27fdd9bc-1d18-4493-b7a6-f49473c74ef2"). InnerVolumeSpecName "kube-api-access-vwddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:40:38.343046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.342962 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:38.343046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.342993 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwddk\" (UniqueName: \"kubernetes.io/projected/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-kube-api-access-vwddk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:38.343046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:38.343003 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27fdd9bc-1d18-4493-b7a6-f49473c74ef2-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:40:39.045437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:39.045404 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" Oct 31 21:40:39.045620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:39.045407 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg" event={"ID":"27fdd9bc-1d18-4493-b7a6-f49473c74ef2","Type":"ContainerDied","Data":"e628a4ebecf014b321ce63d2e42081a841dd38f2e546f72b5e85f9fd1b2fe031"} Oct 31 21:40:39.045620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:39.045538 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e628a4ebecf014b321ce63d2e42081a841dd38f2e546f72b5e85f9fd1b2fe031" Oct 31 21:40:56.365624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:56.365544 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt"] Oct 31 21:40:56.378388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:56.378357 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07afdd-pbzjt"] Oct 31 21:40:57.986161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:40:57.986127 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8f38e2-abab-4b78-8298-f4318afd5eba" path="/var/lib/kubelet/pods/cc8f38e2-abab-4b78-8298-f4318afd5eba/volumes" Oct 31 21:41:00.139890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:00.139855 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:41:00.150369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:00.150345 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:41:01.310672 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:41:01.310641 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:41:05.069404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:05.069366 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg"] Oct 31 21:41:05.086837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:05.086795 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-89d4a2-fxnjg"] Oct 31 21:41:05.986253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:05.986221 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fdd9bc-1d18-4493-b7a6-f49473c74ef2" path="/var/lib/kubelet/pods/27fdd9bc-1d18-4493-b7a6-f49473c74ef2/volumes" Oct 31 21:41:12.985282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:12.985248 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq"] Oct 31 21:41:12.985835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:12.985820 4294 cpu_manager.go:414] "containerMap: removing container" podUID="27fdd9bc-1d18-4493-b7a6-f49473c74ef2" containerName="revoke-keys" Oct 31 21:41:12.985896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:12.985837 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fdd9bc-1d18-4493-b7a6-f49473c74ef2" containerName="revoke-keys" Oct 31 21:41:12.985945 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:12.985936 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fdd9bc-1d18-4493-b7a6-f49473c74ef2" containerName="revoke-keys" Oct 31 21:41:12.990923 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:12.990907 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.007499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.007476 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq"] Oct 31 21:41:13.015763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.015741 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:41:13.015951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.015899 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:41:13.082896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.082863 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.083096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.083078 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmq5\" (UniqueName: \"kubernetes.io/projected/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-kube-api-access-kbmq5\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.083208 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.083190 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.184808 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.184779 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmq5\" (UniqueName: \"kubernetes.io/projected/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-kube-api-access-kbmq5\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.184966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.184862 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.184966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.184916 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.185628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.185608 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.187165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.187142 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.207934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.207909 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmq5\" (UniqueName: \"kubernetes.io/projected/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-kube-api-access-kbmq5\") pod \"maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.304581 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.304505 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:13.630011 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:13.629977 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq"] Oct 31 21:41:13.630550 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:41:13.630522 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf4fe58_d9ba_46c3_9a70_f0e7ecb2843f.slice/crio-b9bc06ca25bbfdbe6b9965972c56b6b78550fa430adbcfd019f7aa5dd8cc2b80 WatchSource:0}: Error finding container b9bc06ca25bbfdbe6b9965972c56b6b78550fa430adbcfd019f7aa5dd8cc2b80: Status 404 returned error can't find the container with id b9bc06ca25bbfdbe6b9965972c56b6b78550fa430adbcfd019f7aa5dd8cc2b80 Oct 31 21:41:14.282474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:14.282441 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" event={"ID":"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f","Type":"ContainerStarted","Data":"3760c0678ebfefbca5699a5e8605ac8e915f5c05791df8d953733a12a1a8fc94"} Oct 31 21:41:14.282851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:14.282500 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" event={"ID":"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f","Type":"ContainerStarted","Data":"b9bc06ca25bbfdbe6b9965972c56b6b78550fa430adbcfd019f7aa5dd8cc2b80"} Oct 31 21:41:14.323258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:14.323210 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" podStartSLOduration=2.323197401 podStartE2EDuration="2.323197401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:14.322339716 +0000 UTC m=+1514.447446947" watchObservedRunningTime="2026-10-31 21:41:14.323197401 +0000 UTC m=+1514.448304632" Oct 31 21:41:15.289717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:15.289681 4294 generic.go:356] "Generic (PLEG): container finished" podID="1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" containerID="3760c0678ebfefbca5699a5e8605ac8e915f5c05791df8d953733a12a1a8fc94" exitCode=0 Oct 31 21:41:15.290258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:15.289762 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" event={"ID":"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f","Type":"ContainerDied","Data":"3760c0678ebfefbca5699a5e8605ac8e915f5c05791df8d953733a12a1a8fc94"} Oct 31 21:41:16.473725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.473695 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:16.581738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.581655 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-kube-api-access-kbmq5\" (UniqueName: \"kubernetes.io/projected/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-kube-api-access-kbmq5\") pod \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " Oct 31 21:41:16.581738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.581724 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-openshift-ca\") pod \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " Oct 31 21:41:16.581963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.581857 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-xks-ca\") pod \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\" (UID: \"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f\") " Oct 31 21:41:16.582104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.582061 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-openshift-ca" pod "1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" (UID: "1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:16.582539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.582517 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:16.583839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.583816 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-xks-ca" pod "1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" (UID: "1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:16.583908 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.583843 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-kube-api-access-kbmq5" pod "1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" (UID: "1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f"). InnerVolumeSpecName "kube-api-access-kbmq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:16.684221 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.684184 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:16.684221 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:16.684217 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmq5\" (UniqueName: \"kubernetes.io/projected/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f-kube-api-access-kbmq5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:17.304182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:17.304149 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" event={"ID":"1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f","Type":"ContainerDied","Data":"b9bc06ca25bbfdbe6b9965972c56b6b78550fa430adbcfd019f7aa5dd8cc2b80"} Oct 31 21:41:17.304414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:17.304210 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9bc06ca25bbfdbe6b9965972c56b6b78550fa430adbcfd019f7aa5dd8cc2b80" Oct 31 21:41:17.304414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:17.304157 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq" Oct 31 21:41:27.999736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:27.999702 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh"] Oct 31 21:41:28.000363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:27.999901 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" podUID="de237056-17be-4ed5-ad5f-03a2d637d066" containerName="maas-api" containerID="cri-o://4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860" gracePeriod=30 Oct 31 21:41:28.318883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.318854 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:41:28.354466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.354433 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de237056-17be-4ed5-ad5f-03a2d637d066-kube-api-access-9tbnp\" (UniqueName: \"kubernetes.io/projected/de237056-17be-4ed5-ad5f-03a2d637d066-kube-api-access-9tbnp\") pod \"de237056-17be-4ed5-ad5f-03a2d637d066\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " Oct 31 21:41:28.354622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.354568 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-maas-api-tls\") pod \"de237056-17be-4ed5-ad5f-03a2d637d066\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " Oct 31 21:41:28.354622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.354603 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-metrics-certs\") pod \"de237056-17be-4ed5-ad5f-03a2d637d066\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " Oct 31 21:41:28.354741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.354713 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de237056-17be-4ed5-ad5f-03a2d637d066-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de237056-17be-4ed5-ad5f-03a2d637d066-openshift-service-ca\") pod \"de237056-17be-4ed5-ad5f-03a2d637d066\" (UID: \"de237056-17be-4ed5-ad5f-03a2d637d066\") " Oct 31 21:41:28.355120 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.355090 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de237056-17be-4ed5-ad5f-03a2d637d066-openshift-service-ca" pod "de237056-17be-4ed5-ad5f-03a2d637d066" (UID: "de237056-17be-4ed5-ad5f-03a2d637d066"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:28.355431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.355409 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de237056-17be-4ed5-ad5f-03a2d637d066-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:28.356517 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.356496 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de237056-17be-4ed5-ad5f-03a2d637d066-kube-api-access-9tbnp" pod "de237056-17be-4ed5-ad5f-03a2d637d066" (UID: "de237056-17be-4ed5-ad5f-03a2d637d066"). InnerVolumeSpecName "kube-api-access-9tbnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:28.356616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.356593 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-metrics-certs" pod "de237056-17be-4ed5-ad5f-03a2d637d066" (UID: "de237056-17be-4ed5-ad5f-03a2d637d066"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:28.356878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.356857 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-maas-api-tls" pod "de237056-17be-4ed5-ad5f-03a2d637d066" (UID: "de237056-17be-4ed5-ad5f-03a2d637d066"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:28.394568 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.394538 4294 generic.go:356] "Generic (PLEG): container finished" podID="de237056-17be-4ed5-ad5f-03a2d637d066" containerID="4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860" exitCode=0 Oct 31 21:41:28.394734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.394592 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" Oct 31 21:41:28.394734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.394622 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" event={"ID":"de237056-17be-4ed5-ad5f-03a2d637d066","Type":"ContainerDied","Data":"4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860"} Oct 31 21:41:28.394734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.394680 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh" event={"ID":"de237056-17be-4ed5-ad5f-03a2d637d066","Type":"ContainerDied","Data":"4967043b124d8ff0b6213edebf581ef30cb46a66b913ac3cebb2b19657d43120"} Oct 31 21:41:28.394734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.394708 4294 scope.go:122] "RemoveContainer" containerID="4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860" Oct 31 21:41:28.408938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.408916 4294 scope.go:122] "RemoveContainer" containerID="4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860" Oct 31 21:41:28.410201 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:41:28.410170 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860\": container with ID starting with 4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860 not found: ID does not exist" containerID="4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860" Oct 31 21:41:28.410895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.410213 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860"} err="failed to get container status \"4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860\": rpc error: code = NotFound desc = could not find container \"4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860\": container with ID starting with 4f5c4c7c616a2cb51700a7670611064fa514cff4adc0443c839fc7c99ce1c860 not found: ID does not exist" Oct 31 21:41:28.448676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.448649 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh"] Oct 31 21:41:28.454679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.454656 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c40fc3-67475bf878-7z8hh"] Oct 31 21:41:28.457465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.457444 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:28.457465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.457466 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de237056-17be-4ed5-ad5f-03a2d637d066-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:28.457605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:28.457475 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbnp\" (UniqueName: \"kubernetes.io/projected/de237056-17be-4ed5-ad5f-03a2d637d066-kube-api-access-9tbnp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:29.985569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:29.985525 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de237056-17be-4ed5-ad5f-03a2d637d066" path="/var/lib/kubelet/pods/de237056-17be-4ed5-ad5f-03a2d637d066/volumes" Oct 31 21:41:31.310461 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:41:31.310423 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:41:32.579437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.579400 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww"] Oct 31 21:41:32.579972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.579959 4294 cpu_manager.go:414] "containerMap: removing container" podUID="de237056-17be-4ed5-ad5f-03a2d637d066" containerName="maas-api" Oct 31 21:41:32.580015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.579973 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de237056-17be-4ed5-ad5f-03a2d637d066" containerName="maas-api" Oct 31 21:41:32.580015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.579996 4294 cpu_manager.go:414] "containerMap: removing container" podUID="1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" containerName="revoke-keys" Oct 31 21:41:32.580015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.580001 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" containerName="revoke-keys" Oct 31 21:41:32.580119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.580109 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="de237056-17be-4ed5-ad5f-03a2d637d066" containerName="maas-api" Oct 31 21:41:32.580152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.580124 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" containerName="revoke-keys" Oct 31 21:41:32.584719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.584702 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.599939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.599431 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:41:32.599939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.599454 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:41:32.602756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.602721 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww"] Oct 31 21:41:32.625729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.625676 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr8zg\" (UniqueName: \"kubernetes.io/projected/cb770b44-cc79-4ed3-b534-d0d11e867eee-kube-api-access-cr8zg\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.625939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.625873 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.626048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.625935 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.727510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.727467 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.727706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.727659 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr8zg\" (UniqueName: \"kubernetes.io/projected/cb770b44-cc79-4ed3-b534-d0d11e867eee-kube-api-access-cr8zg\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.727819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.727805 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.728111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.728091 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.730078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.730062 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.751144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.751120 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr8zg\" (UniqueName: \"kubernetes.io/projected/cb770b44-cc79-4ed3-b534-d0d11e867eee-kube-api-access-cr8zg\") pod \"maas-api-revoke-sub-432f3459-bkwww\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:32.898284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:32.898206 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:33.199717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:33.199684 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww"] Oct 31 21:41:33.200831 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:41:33.200807 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb770b44_cc79_4ed3_b534_d0d11e867eee.slice/crio-248fdc297f92f504b5cb645078c39cb9980bb8b36753e4ea4e986f61b9689b77 WatchSource:0}: Error finding container 248fdc297f92f504b5cb645078c39cb9980bb8b36753e4ea4e986f61b9689b77: Status 404 returned error can't find the container with id 248fdc297f92f504b5cb645078c39cb9980bb8b36753e4ea4e986f61b9689b77 Oct 31 21:41:33.427848 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:33.427814 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" event={"ID":"cb770b44-cc79-4ed3-b534-d0d11e867eee","Type":"ContainerStarted","Data":"248fdc297f92f504b5cb645078c39cb9980bb8b36753e4ea4e986f61b9689b77"} Oct 31 21:41:34.439743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:34.439647 4294 generic.go:356] "Generic (PLEG): container finished" podID="cb770b44-cc79-4ed3-b534-d0d11e867eee" containerID="ff80d410815a4efc4cfe15b1defeb8b46d6a6ffec2e473b149b844ac1463a84f" exitCode=0 Oct 31 21:41:34.440411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:34.440385 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" event={"ID":"cb770b44-cc79-4ed3-b534-d0d11e867eee","Type":"ContainerDied","Data":"ff80d410815a4efc4cfe15b1defeb8b46d6a6ffec2e473b149b844ac1463a84f"} Oct 31 21:41:35.639992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.639970 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:35.771212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.771184 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-openshift-ca\") pod \"cb770b44-cc79-4ed3-b534-d0d11e867eee\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " Oct 31 21:41:35.771409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.771287 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-xks-ca\") pod \"cb770b44-cc79-4ed3-b534-d0d11e867eee\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " Oct 31 21:41:35.771478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.771462 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb770b44-cc79-4ed3-b534-d0d11e867eee-kube-api-access-cr8zg\" (UniqueName: \"kubernetes.io/projected/cb770b44-cc79-4ed3-b534-d0d11e867eee-kube-api-access-cr8zg\") pod \"cb770b44-cc79-4ed3-b534-d0d11e867eee\" (UID: \"cb770b44-cc79-4ed3-b534-d0d11e867eee\") " Oct 31 21:41:35.771629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.771604 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-openshift-ca" pod "cb770b44-cc79-4ed3-b534-d0d11e867eee" (UID: "cb770b44-cc79-4ed3-b534-d0d11e867eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:35.771982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.771960 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:35.773282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.773256 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-xks-ca" pod "cb770b44-cc79-4ed3-b534-d0d11e867eee" (UID: "cb770b44-cc79-4ed3-b534-d0d11e867eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:35.773443 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.773417 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb770b44-cc79-4ed3-b534-d0d11e867eee-kube-api-access-cr8zg" pod "cb770b44-cc79-4ed3-b534-d0d11e867eee" (UID: "cb770b44-cc79-4ed3-b534-d0d11e867eee"). InnerVolumeSpecName "kube-api-access-cr8zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:35.873585 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.873528 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb770b44-cc79-4ed3-b534-d0d11e867eee-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:35.873585 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:35.873568 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr8zg\" (UniqueName: \"kubernetes.io/projected/cb770b44-cc79-4ed3-b534-d0d11e867eee-kube-api-access-cr8zg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:36.266663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:36.264611 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z"] Oct 31 21:41:36.266663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:36.265241 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" containerID="cri-o://1a0c1329f9eec6ebda50328f755eee3f02f9b85f461a5a57c67ef4851e9b8cd3" gracePeriod=30 Oct 31 21:41:36.458323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:36.457488 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" event={"ID":"cb770b44-cc79-4ed3-b534-d0d11e867eee","Type":"ContainerDied","Data":"248fdc297f92f504b5cb645078c39cb9980bb8b36753e4ea4e986f61b9689b77"} Oct 31 21:41:36.458323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:36.457534 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww" Oct 31 21:41:36.458323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:36.457549 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="248fdc297f92f504b5cb645078c39cb9980bb8b36753e4ea4e986f61b9689b77" Oct 31 21:41:37.151261 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.138815 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq"] Oct 31 21:41:37.173020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.172942 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-752623-ntlmq"] Oct 31 21:41:37.273864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.273828 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6"] Oct 31 21:41:37.274415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.274401 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cb770b44-cc79-4ed3-b534-d0d11e867eee" containerName="revoke-keys" Oct 31 21:41:37.274469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.274416 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb770b44-cc79-4ed3-b534-d0d11e867eee" containerName="revoke-keys" Oct 31 21:41:37.274529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.274520 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb770b44-cc79-4ed3-b534-d0d11e867eee" containerName="revoke-keys" Oct 31 21:41:37.278002 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.277986 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.294488 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.294447 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:41:37.294639 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.294591 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:41:37.296780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.296756 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6"] Oct 31 21:41:37.400138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.400101 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.400312 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.400261 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.400452 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.400432 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdr6w\" (UniqueName: \"kubernetes.io/projected/29111013-cb43-4d28-aa5e-d69847f21ea8-kube-api-access-gdr6w\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.502348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.502281 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.502536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.502455 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdr6w\" (UniqueName: \"kubernetes.io/projected/29111013-cb43-4d28-aa5e-d69847f21ea8-kube-api-access-gdr6w\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.502634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.502618 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.503231 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.503201 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.504710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.504690 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.526811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.526780 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdr6w\" (UniqueName: \"kubernetes.io/projected/29111013-cb43-4d28-aa5e-d69847f21ea8-kube-api-access-gdr6w\") pod \"maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.594730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.594698 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:37.907853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.907823 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6"] Oct 31 21:41:37.910638 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:41:37.910609 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29111013_cb43_4d28_aa5e_d69847f21ea8.slice/crio-7f67f9c07544493d3ade9f0eeae41da6fe9a6b47bf52fab2f9c90f968d9b6842 WatchSource:0}: Error finding container 7f67f9c07544493d3ade9f0eeae41da6fe9a6b47bf52fab2f9c90f968d9b6842: Status 404 returned error can't find the container with id 7f67f9c07544493d3ade9f0eeae41da6fe9a6b47bf52fab2f9c90f968d9b6842 Oct 31 21:41:37.987656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:37.987618 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f" path="/var/lib/kubelet/pods/1bf4fe58-d9ba-46c3-9a70-f0e7ecb2843f/volumes" Oct 31 21:41:38.473515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:38.473423 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" event={"ID":"29111013-cb43-4d28-aa5e-d69847f21ea8","Type":"ContainerStarted","Data":"a35bca9ac7345b8c1e055895fff5eaefcc49891453e21722e627540ae9b34c6b"} Oct 31 21:41:38.473515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:38.473490 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" event={"ID":"29111013-cb43-4d28-aa5e-d69847f21ea8","Type":"ContainerStarted","Data":"7f67f9c07544493d3ade9f0eeae41da6fe9a6b47bf52fab2f9c90f968d9b6842"} Oct 31 21:41:38.517119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:38.517053 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" podStartSLOduration=1.517035321 podStartE2EDuration="1.517035321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:38.514501593 +0000 UTC m=+1538.639608825" watchObservedRunningTime="2026-10-31 21:41:38.517035321 +0000 UTC m=+1538.642142553" Oct 31 21:41:39.481320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:39.481272 4294 generic.go:356] "Generic (PLEG): container finished" podID="29111013-cb43-4d28-aa5e-d69847f21ea8" containerID="a35bca9ac7345b8c1e055895fff5eaefcc49891453e21722e627540ae9b34c6b" exitCode=0 Oct 31 21:41:39.481746 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:39.481333 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" event={"ID":"29111013-cb43-4d28-aa5e-d69847f21ea8","Type":"ContainerDied","Data":"a35bca9ac7345b8c1e055895fff5eaefcc49891453e21722e627540ae9b34c6b"} Oct 31 21:41:40.658640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.658614 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:40.747591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.747512 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29111013-cb43-4d28-aa5e-d69847f21ea8-kube-api-access-gdr6w\" (UniqueName: \"kubernetes.io/projected/29111013-cb43-4d28-aa5e-d69847f21ea8-kube-api-access-gdr6w\") pod \"29111013-cb43-4d28-aa5e-d69847f21ea8\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " Oct 31 21:41:40.747743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.747594 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-xks-ca\") pod \"29111013-cb43-4d28-aa5e-d69847f21ea8\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " Oct 31 21:41:40.747743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.747675 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-openshift-ca\") pod \"29111013-cb43-4d28-aa5e-d69847f21ea8\" (UID: \"29111013-cb43-4d28-aa5e-d69847f21ea8\") " Oct 31 21:41:40.748030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.748005 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-openshift-ca" pod "29111013-cb43-4d28-aa5e-d69847f21ea8" (UID: "29111013-cb43-4d28-aa5e-d69847f21ea8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:40.749604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.749577 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29111013-cb43-4d28-aa5e-d69847f21ea8-kube-api-access-gdr6w" pod "29111013-cb43-4d28-aa5e-d69847f21ea8" (UID: "29111013-cb43-4d28-aa5e-d69847f21ea8"). InnerVolumeSpecName "kube-api-access-gdr6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:40.749689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.749636 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-xks-ca" pod "29111013-cb43-4d28-aa5e-d69847f21ea8" (UID: "29111013-cb43-4d28-aa5e-d69847f21ea8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:40.849725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.849696 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:40.849725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.849721 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdr6w\" (UniqueName: \"kubernetes.io/projected/29111013-cb43-4d28-aa5e-d69847f21ea8-kube-api-access-gdr6w\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:40.849725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:40.849732 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29111013-cb43-4d28-aa5e-d69847f21ea8-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.500215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.500184 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" Oct 31 21:41:41.500415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.500212 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6" event={"ID":"29111013-cb43-4d28-aa5e-d69847f21ea8","Type":"ContainerDied","Data":"7f67f9c07544493d3ade9f0eeae41da6fe9a6b47bf52fab2f9c90f968d9b6842"} Oct 31 21:41:41.500415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.500276 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f67f9c07544493d3ade9f0eeae41da6fe9a6b47bf52fab2f9c90f968d9b6842" Oct 31 21:41:41.504007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.503973 4294 generic.go:356] "Generic (PLEG): container finished" podID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerID="1a0c1329f9eec6ebda50328f755eee3f02f9b85f461a5a57c67ef4851e9b8cd3" exitCode=0 Oct 31 21:41:41.504225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.504100 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" event={"ID":"a82102cf-6d60-4be1-82e3-40c67d14215d","Type":"ContainerDied","Data":"1a0c1329f9eec6ebda50328f755eee3f02f9b85f461a5a57c67ef4851e9b8cd3"} Oct 31 21:41:41.504225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.504152 4294 scope.go:122] "RemoveContainer" containerID="aa1f06fb7b0901b7673cfb3dc0c3daee585338e445609dab2338cace5fd5cc67" Oct 31 21:41:41.533075 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.533053 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:41:41.662028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.661998 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-envoy\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662420 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662083 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-kube-api-access-77hm2\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-kube-api-access-77hm2\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662420 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662120 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-certs\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662420 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662248 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-socket\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662420 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662296 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-credential-socket\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662433 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-certs" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:41.662649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662486 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-podinfo\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662525 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-token\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-token\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662554 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-data\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662838 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662691 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82102cf-6d60-4be1-82e3-40c67d14215d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82102cf-6d60-4be1-82e3-40c67d14215d-istiod-ca-cert\") pod \"a82102cf-6d60-4be1-82e3-40c67d14215d\" (UID: \"a82102cf-6d60-4be1-82e3-40c67d14215d\") " Oct 31 21:41:41.662838 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662550 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-credential-socket" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:41.662838 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.662750 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-socket" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:41.663189 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.663165 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-data" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:41.663330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.663289 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82102cf-6d60-4be1-82e3-40c67d14215d-istiod-ca-cert" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:41.663484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.663466 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.663557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.663492 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.663557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.663506 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.663557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.663519 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.664527 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.664503 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-envoy" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:41:41.664787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.664768 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-token" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:41.664854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.664795 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-kube-api-access-77hm2" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "kube-api-access-77hm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:41.665064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.665048 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-podinfo" pod "a82102cf-6d60-4be1-82e3-40c67d14215d" (UID: "a82102cf-6d60-4be1-82e3-40c67d14215d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:41:41.765313 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.765270 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.765473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.765326 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.765473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.765338 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a82102cf-6d60-4be1-82e3-40c67d14215d-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.765473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.765345 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a82102cf-6d60-4be1-82e3-40c67d14215d-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:41.765473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:41.765354 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77hm2\" (UniqueName: \"kubernetes.io/projected/a82102cf-6d60-4be1-82e3-40c67d14215d-kube-api-access-77hm2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:42.520135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.520088 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" event={"ID":"a82102cf-6d60-4be1-82e3-40c67d14215d","Type":"ContainerDied","Data":"debb535211aa259da83d880a2cf31d4a7d5eba18b9e05b7907ee8bea9775d859"} Oct 31 21:41:42.520342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.520172 4294 scope.go:122] "RemoveContainer" containerID="1a0c1329f9eec6ebda50328f755eee3f02f9b85f461a5a57c67ef4851e9b8cd3" Oct 31 21:41:42.520438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.520420 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z" Oct 31 21:41:42.575225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.575186 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z"] Oct 31 21:41:42.587855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.587818 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c40fc3-openshift-default-fcbcf5cb5-7dm2z"] Oct 31 21:41:42.939756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.939677 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx"] Oct 31 21:41:42.940250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940236 4294 cpu_manager.go:414] "containerMap: removing container" podUID="29111013-cb43-4d28-aa5e-d69847f21ea8" containerName="revoke-keys" Oct 31 21:41:42.940366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940252 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29111013-cb43-4d28-aa5e-d69847f21ea8" containerName="revoke-keys" Oct 31 21:41:42.940366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940272 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" Oct 31 21:41:42.940366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940277 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" Oct 31 21:41:42.940366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940295 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" Oct 31 21:41:42.940366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940313 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" Oct 31 21:41:42.940530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940427 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="29111013-cb43-4d28-aa5e-d69847f21ea8" containerName="revoke-keys" Oct 31 21:41:42.940530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940439 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" Oct 31 21:41:42.940530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.940449 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" containerName="istio-proxy" Oct 31 21:41:42.945050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.945028 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:42.961522 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.961497 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:41:42.961673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.961621 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:41:42.962899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:42.962875 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx"] Oct 31 21:41:43.083938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.083908 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdggm\" (UniqueName: \"kubernetes.io/projected/fbdbe909-ff3e-4564-acdd-56592436dddf-kube-api-access-pdggm\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.084135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.083986 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.084203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.084178 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.185998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.185961 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdggm\" (UniqueName: \"kubernetes.io/projected/fbdbe909-ff3e-4564-acdd-56592436dddf-kube-api-access-pdggm\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.186182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.186037 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.186182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.186119 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.186669 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.186640 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.188438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.188416 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.209429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.209379 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdggm\" (UniqueName: \"kubernetes.io/projected/fbdbe909-ff3e-4564-acdd-56592436dddf-kube-api-access-pdggm\") pod \"maas-api-revoke-sub-54dbb230-76xcx\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.262013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.261989 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:43.573839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.573810 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx"] Oct 31 21:41:43.575337 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:41:43.575282 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdbe909_ff3e_4564_acdd_56592436dddf.slice/crio-9f8385bce81ca39a8d91a62706f9509bd024d6ce3ec3154f9d96a764eeefdbe9 WatchSource:0}: Error finding container 9f8385bce81ca39a8d91a62706f9509bd024d6ce3ec3154f9d96a764eeefdbe9: Status 404 returned error can't find the container with id 9f8385bce81ca39a8d91a62706f9509bd024d6ce3ec3154f9d96a764eeefdbe9 Oct 31 21:41:43.984920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:43.984886 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82102cf-6d60-4be1-82e3-40c67d14215d" path="/var/lib/kubelet/pods/a82102cf-6d60-4be1-82e3-40c67d14215d/volumes" Oct 31 21:41:44.538855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:44.538824 4294 generic.go:356] "Generic (PLEG): container finished" podID="fbdbe909-ff3e-4564-acdd-56592436dddf" containerID="89ed7b215920f1652cc652f8902c28c6883c6b567b94bded28b086b0637d34be" exitCode=0 Oct 31 21:41:44.539005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:44.538993 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" event={"ID":"fbdbe909-ff3e-4564-acdd-56592436dddf","Type":"ContainerDied","Data":"89ed7b215920f1652cc652f8902c28c6883c6b567b94bded28b086b0637d34be"} Oct 31 21:41:44.539076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:44.539037 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" event={"ID":"fbdbe909-ff3e-4564-acdd-56592436dddf","Type":"ContainerStarted","Data":"9f8385bce81ca39a8d91a62706f9509bd024d6ce3ec3154f9d96a764eeefdbe9"} Oct 31 21:41:45.697201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.697179 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:45.719933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.719908 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-xks-ca\") pod \"fbdbe909-ff3e-4564-acdd-56592436dddf\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " Oct 31 21:41:45.720034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.720021 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-openshift-ca\") pod \"fbdbe909-ff3e-4564-acdd-56592436dddf\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " Oct 31 21:41:45.720130 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.720118 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbdbe909-ff3e-4564-acdd-56592436dddf-kube-api-access-pdggm\" (UniqueName: \"kubernetes.io/projected/fbdbe909-ff3e-4564-acdd-56592436dddf-kube-api-access-pdggm\") pod \"fbdbe909-ff3e-4564-acdd-56592436dddf\" (UID: \"fbdbe909-ff3e-4564-acdd-56592436dddf\") " Oct 31 21:41:45.720358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.720324 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-openshift-ca" pod "fbdbe909-ff3e-4564-acdd-56592436dddf" (UID: "fbdbe909-ff3e-4564-acdd-56592436dddf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:41:45.720534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.720519 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:45.721926 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.721904 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-xks-ca" pod "fbdbe909-ff3e-4564-acdd-56592436dddf" (UID: "fbdbe909-ff3e-4564-acdd-56592436dddf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:41:45.722053 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.722034 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbdbe909-ff3e-4564-acdd-56592436dddf-kube-api-access-pdggm" pod "fbdbe909-ff3e-4564-acdd-56592436dddf" (UID: "fbdbe909-ff3e-4564-acdd-56592436dddf"). InnerVolumeSpecName "kube-api-access-pdggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:41:45.821861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.821782 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdggm\" (UniqueName: \"kubernetes.io/projected/fbdbe909-ff3e-4564-acdd-56592436dddf-kube-api-access-pdggm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:45.821861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:45.821810 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbdbe909-ff3e-4564-acdd-56592436dddf-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:41:46.554490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:46.554456 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" Oct 31 21:41:46.554701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:46.554489 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx" event={"ID":"fbdbe909-ff3e-4564-acdd-56592436dddf","Type":"ContainerDied","Data":"9f8385bce81ca39a8d91a62706f9509bd024d6ce3ec3154f9d96a764eeefdbe9"} Oct 31 21:41:46.554753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:46.554735 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f8385bce81ca39a8d91a62706f9509bd024d6ce3ec3154f9d96a764eeefdbe9" Oct 31 21:41:57.638242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.638198 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd"] Oct 31 21:41:57.638805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.638789 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fbdbe909-ff3e-4564-acdd-56592436dddf" containerName="revoke-keys" Oct 31 21:41:57.638852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.638807 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbdbe909-ff3e-4564-acdd-56592436dddf" containerName="revoke-keys" Oct 31 21:41:57.638926 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.638917 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbdbe909-ff3e-4564-acdd-56592436dddf" containerName="revoke-keys" Oct 31 21:41:57.641721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.641705 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.659868 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.659843 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd"] Oct 31 21:41:57.667074 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.667050 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:41:57.667183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.667089 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:41:57.741033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.741003 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.741238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.741220 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.741278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.741263 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gxx5\" (UniqueName: \"kubernetes.io/projected/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-kube-api-access-5gxx5\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.842662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.842626 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.842864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.842680 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gxx5\" (UniqueName: \"kubernetes.io/projected/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-kube-api-access-5gxx5\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.842864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.842763 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.843426 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.843388 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.845248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.845227 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.865755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.865723 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gxx5\" (UniqueName: \"kubernetes.io/projected/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-kube-api-access-5gxx5\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:57.956675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:57.956604 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:41:58.291419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:58.291221 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd"] Oct 31 21:41:58.296263 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:41:58.296219 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51997c34_e3a4_4981_8fab_6e4f0ccd66ce.slice/crio-a9e1916eeb1bfea77d436871d5688833b66a3cd9e70363d9ea1692749413854a WatchSource:0}: Error finding container a9e1916eeb1bfea77d436871d5688833b66a3cd9e70363d9ea1692749413854a: Status 404 returned error can't find the container with id a9e1916eeb1bfea77d436871d5688833b66a3cd9e70363d9ea1692749413854a Oct 31 21:41:58.716767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:58.716729 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" event={"ID":"51997c34-e3a4-4981-8fab-6e4f0ccd66ce","Type":"ContainerStarted","Data":"af7127dc41d80fdfcd8d2b5c0a4464dfc3c2179a5c7f4f799eaa2573b5d539ed"} Oct 31 21:41:58.717129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:58.716788 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" event={"ID":"51997c34-e3a4-4981-8fab-6e4f0ccd66ce","Type":"ContainerStarted","Data":"a9e1916eeb1bfea77d436871d5688833b66a3cd9e70363d9ea1692749413854a"} Oct 31 21:41:58.758526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:58.758478 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" podStartSLOduration=1.758465369 podStartE2EDuration="1.758465369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:41:58.756405616 +0000 UTC m=+1558.881512848" watchObservedRunningTime="2026-10-31 21:41:58.758465369 +0000 UTC m=+1558.883572646" Oct 31 21:41:59.729966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:59.729930 4294 generic.go:356] "Generic (PLEG): container finished" podID="51997c34-e3a4-4981-8fab-6e4f0ccd66ce" containerID="af7127dc41d80fdfcd8d2b5c0a4464dfc3c2179a5c7f4f799eaa2573b5d539ed" exitCode=0 Oct 31 21:41:59.730511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:41:59.730004 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" event={"ID":"51997c34-e3a4-4981-8fab-6e4f0ccd66ce","Type":"ContainerDied","Data":"af7127dc41d80fdfcd8d2b5c0a4464dfc3c2179a5c7f4f799eaa2573b5d539ed"} Oct 31 21:42:00.931968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:00.931941 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:42:01.112444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.112351 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-xks-ca\") pod \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " Oct 31 21:42:01.112444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.112421 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-openshift-ca\") pod \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " Oct 31 21:42:01.112670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.112516 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-kube-api-access-5gxx5\" (UniqueName: \"kubernetes.io/projected/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-kube-api-access-5gxx5\") pod \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\" (UID: \"51997c34-e3a4-4981-8fab-6e4f0ccd66ce\") " Oct 31 21:42:01.114014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.113980 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-openshift-ca" pod "51997c34-e3a4-4981-8fab-6e4f0ccd66ce" (UID: "51997c34-e3a4-4981-8fab-6e4f0ccd66ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:01.115343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.115283 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-kube-api-access-5gxx5" pod "51997c34-e3a4-4981-8fab-6e4f0ccd66ce" (UID: "51997c34-e3a4-4981-8fab-6e4f0ccd66ce"). InnerVolumeSpecName "kube-api-access-5gxx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:01.115984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.115959 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-xks-ca" pod "51997c34-e3a4-4981-8fab-6e4f0ccd66ce" (UID: "51997c34-e3a4-4981-8fab-6e4f0ccd66ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:01.217328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.214761 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:42:01.217328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.214807 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:42:01.217328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.214825 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gxx5\" (UniqueName: \"kubernetes.io/projected/51997c34-e3a4-4981-8fab-6e4f0ccd66ce-kube-api-access-5gxx5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:42:01.359766 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:42:01.358257 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:42:01.756422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.756395 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" Oct 31 21:42:01.756422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.756410 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd" event={"ID":"51997c34-e3a4-4981-8fab-6e4f0ccd66ce","Type":"ContainerDied","Data":"a9e1916eeb1bfea77d436871d5688833b66a3cd9e70363d9ea1692749413854a"} Oct 31 21:42:01.756667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:01.756460 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9e1916eeb1bfea77d436871d5688833b66a3cd9e70363d9ea1692749413854a" Oct 31 21:42:06.768632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:06.768597 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6"] Oct 31 21:42:06.781893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:06.781812 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9bbfc5-gtsq6"] Oct 31 21:42:07.985247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:07.985203 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29111013-cb43-4d28-aa5e-d69847f21ea8" path="/var/lib/kubelet/pods/29111013-cb43-4d28-aa5e-d69847f21ea8/volumes" Oct 31 21:42:31.313474 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:42:31.313433 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:42:35.188201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.188169 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686"] Oct 31 21:42:35.188733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.188422 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" podUID="069ea2b3-55cc-45b7-9a52-95897f5c59cc" containerName="maas-api" containerID="cri-o://3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595" gracePeriod=30 Oct 31 21:42:35.783324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.776659 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:42:35.798323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.798036 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069ea2b3-55cc-45b7-9a52-95897f5c59cc-kube-api-access-xsp9w\" (UniqueName: \"kubernetes.io/projected/069ea2b3-55cc-45b7-9a52-95897f5c59cc-kube-api-access-xsp9w\") pod \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " Oct 31 21:42:35.798323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.798130 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-metrics-certs\") pod \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " Oct 31 21:42:35.798323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.798221 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls\") pod \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " Oct 31 21:42:35.798323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.798277 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069ea2b3-55cc-45b7-9a52-95897f5c59cc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069ea2b3-55cc-45b7-9a52-95897f5c59cc-openshift-service-ca\") pod \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\" (UID: \"069ea2b3-55cc-45b7-9a52-95897f5c59cc\") " Oct 31 21:42:35.819572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.819506 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069ea2b3-55cc-45b7-9a52-95897f5c59cc-openshift-service-ca" pod "069ea2b3-55cc-45b7-9a52-95897f5c59cc" (UID: "069ea2b3-55cc-45b7-9a52-95897f5c59cc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:42:35.845240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.845160 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-metrics-certs" pod "069ea2b3-55cc-45b7-9a52-95897f5c59cc" (UID: "069ea2b3-55cc-45b7-9a52-95897f5c59cc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:35.845919 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.845868 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls" pod "069ea2b3-55cc-45b7-9a52-95897f5c59cc" (UID: "069ea2b3-55cc-45b7-9a52-95897f5c59cc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:42:35.846225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.846186 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069ea2b3-55cc-45b7-9a52-95897f5c59cc-kube-api-access-xsp9w" pod "069ea2b3-55cc-45b7-9a52-95897f5c59cc" (UID: "069ea2b3-55cc-45b7-9a52-95897f5c59cc"). InnerVolumeSpecName "kube-api-access-xsp9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:42:35.900681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.900643 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:42:35.900681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.900685 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/069ea2b3-55cc-45b7-9a52-95897f5c59cc-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:42:35.901006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.900702 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsp9w\" (UniqueName: \"kubernetes.io/projected/069ea2b3-55cc-45b7-9a52-95897f5c59cc-kube-api-access-xsp9w\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:42:35.901006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:35.900715 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/069ea2b3-55cc-45b7-9a52-95897f5c59cc-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:42:36.014157 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.014122 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd"] Oct 31 21:42:36.018419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.018390 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc0d65-t9cpd"] Oct 31 21:42:36.022509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.022484 4294 generic.go:356] "Generic (PLEG): container finished" podID="069ea2b3-55cc-45b7-9a52-95897f5c59cc" containerID="3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595" exitCode=0 Oct 31 21:42:36.022695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.022680 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" event={"ID":"069ea2b3-55cc-45b7-9a52-95897f5c59cc","Type":"ContainerDied","Data":"3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595"} Oct 31 21:42:36.022749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.022718 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" event={"ID":"069ea2b3-55cc-45b7-9a52-95897f5c59cc","Type":"ContainerDied","Data":"091c175b2003e945d90686e47c0a07b488b19243b4190274a05cf913af52ebbb"} Oct 31 21:42:36.022749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.022740 4294 scope.go:122] "RemoveContainer" containerID="3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595" Oct 31 21:42:36.022858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.022847 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686" Oct 31 21:42:36.036629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.036612 4294 scope.go:122] "RemoveContainer" containerID="3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595" Oct 31 21:42:36.037255 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:42:36.037224 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595\": container with ID starting with 3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595 not found: ID does not exist" containerID="3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595" Oct 31 21:42:36.037395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.037265 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595"} err="failed to get container status \"3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595\": rpc error: code = NotFound desc = could not find container \"3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595\": container with ID starting with 3d982410c731f7c906e0477c97ff9d1d33bd2995bd977969f8f21360c2b81595 not found: ID does not exist" Oct 31 21:42:36.078862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.078837 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686"] Oct 31 21:42:36.085914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:36.085888 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc0d65-66b6cbdcc5-jh686"] Oct 31 21:42:38.103325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:38.103067 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069ea2b3-55cc-45b7-9a52-95897f5c59cc" path="/var/lib/kubelet/pods/069ea2b3-55cc-45b7-9a52-95897f5c59cc/volumes" Oct 31 21:42:38.116373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:42:38.104216 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51997c34-e3a4-4981-8fab-6e4f0ccd66ce" path="/var/lib/kubelet/pods/51997c34-e3a4-4981-8fab-6e4f0ccd66ce/volumes" Oct 31 21:43:01.310453 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:01.310418 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:43:14.988807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:14.988771 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr"] Oct 31 21:43:14.989241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:14.988960 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" containerID="cri-o://c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143" gracePeriod=30 Oct 31 21:43:16.092602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:16.092563 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6"] Oct 31 21:43:16.096074 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:16.092812 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" containerID="cri-o://6ecc70ced6671b2222b244b451be05401c6012afef6612170480d8324507757b" gracePeriod=30 Oct 31 21:43:20.282384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.282357 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:43:20.393754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.393719 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-socket\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.393935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.393771 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-envoy\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.393935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.393800 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-credential-socket\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.393935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.393841 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-data\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.394098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.393962 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-token\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.394098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394013 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istiod-ca-cert\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.394098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394089 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-certs\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.394295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394100 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-socket" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:20.394295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394123 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-credential-socket" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:20.394295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394250 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-kube-api-access-969dz\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-kube-api-access-969dz\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.394492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394357 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-certs" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:20.394492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394424 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-podinfo\") pod \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\" (UID: \"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd\") " Oct 31 21:43:20.394492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394451 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-data" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:20.394690 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.394668 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istiod-ca-cert" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:20.395021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.395002 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.395112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.395025 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.395112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.395039 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.395112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.395052 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.395112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.395063 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.397103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.397066 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-envoy" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:20.397213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.397110 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-token" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:20.397278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.397232 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-podinfo" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:43:20.397437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.397414 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-kube-api-access-969dz" pod "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" (UID: "9b9fc8b4-6465-412a-84d8-3cef75fb8ffd"). InnerVolumeSpecName "kube-api-access-969dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:20.452737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.452706 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc"] Oct 31 21:43:20.452972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.452948 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" containerID="cri-o://e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a" gracePeriod=30 Oct 31 21:43:20.472127 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.472086 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf"] Oct 31 21:43:20.472580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.472393 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" containerID="cri-o://30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b" gracePeriod=30 Oct 31 21:43:20.488207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.488173 4294 generic.go:356] "Generic (PLEG): container finished" podID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerID="c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143" exitCode=0 Oct 31 21:43:20.488419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.488396 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" event={"ID":"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd","Type":"ContainerDied","Data":"c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143"} Oct 31 21:43:20.488503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.488454 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" event={"ID":"9b9fc8b4-6465-412a-84d8-3cef75fb8ffd","Type":"ContainerDied","Data":"096e67911d2e7ecd6efbb7c27fb53e169c3458edc88c30e99f10589bd0342b99"} Oct 31 21:43:20.488503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.488484 4294 scope.go:122] "RemoveContainer" containerID="c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143" Oct 31 21:43:20.488674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.488656 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr" Oct 31 21:43:20.498106 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.498084 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.498205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.498109 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969dz\" (UniqueName: \"kubernetes.io/projected/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-kube-api-access-969dz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.498205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.498124 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.498205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.498136 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:20.506434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.506409 4294 scope.go:122] "RemoveContainer" containerID="c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143" Oct 31 21:43:20.506744 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:20.506715 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143\": container with ID starting with c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143 not found: ID does not exist" containerID="c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143" Oct 31 21:43:20.506869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.506757 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143"} err="failed to get container status \"c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143\": rpc error: code = NotFound desc = could not find container \"c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143\": container with ID starting with c584a6729cde7fe0c6f971990a24b4af6c2878b2e940642348702347ffdd3143 not found: ID does not exist" Oct 31 21:43:20.621532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.621496 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr"] Oct 31 21:43:20.621687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:20.621554 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-ckdqr"] Oct 31 21:43:21.495576 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.495543 4294 generic.go:356] "Generic (PLEG): container finished" podID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerID="6ecc70ced6671b2222b244b451be05401c6012afef6612170480d8324507757b" exitCode=0 Oct 31 21:43:21.495983 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.495614 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" event={"ID":"4297b98b-9e10-4e72-a322-3e33984deb9e","Type":"ContainerDied","Data":"6ecc70ced6671b2222b244b451be05401c6012afef6612170480d8324507757b"} Oct 31 21:43:21.561772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.561739 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Oct 31 21:43:21.561940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.561807 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Oct 31 21:43:21.790296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.790270 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:43:21.922852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.922813 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-podinfo\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923053 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.922895 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-data\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923053 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.922999 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-token\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-token\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923053 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923035 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-envoy\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923069 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-kube-api-access-ngvwz\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-kube-api-access-ngvwz\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923181 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-certs\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923194 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-data" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:21.923423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923352 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4297b98b-9e10-4e72-a322-3e33984deb9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297b98b-9e10-4e72-a322-3e33984deb9e-istiod-ca-cert\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923403 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-credential-socket\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923470 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-socket\") pod \"4297b98b-9e10-4e72-a322-3e33984deb9e\" (UID: \"4297b98b-9e10-4e72-a322-3e33984deb9e\") " Oct 31 21:43:21.923523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923476 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-certs" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:21.923749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923724 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-credential-socket" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:21.923815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923740 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4297b98b-9e10-4e72-a322-3e33984deb9e-istiod-ca-cert" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:21.923868 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.923837 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-socket" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:21.924132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.924113 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:21.924192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.924139 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:21.924192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.924152 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4297b98b-9e10-4e72-a322-3e33984deb9e-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:21.924192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.924167 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:21.924192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.924182 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:21.925217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.925198 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-podinfo" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:43:21.925357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.925339 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-kube-api-access-ngvwz" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "kube-api-access-ngvwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:21.925676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.925652 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-token" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:21.925676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.925670 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-envoy" pod "4297b98b-9e10-4e72-a322-3e33984deb9e" (UID: "4297b98b-9e10-4e72-a322-3e33984deb9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:21.985731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:21.985701 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" path="/var/lib/kubelet/pods/9b9fc8b4-6465-412a-84d8-3cef75fb8ffd/volumes" Oct 31 21:43:22.025707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.025620 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:22.025707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.025660 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:22.025707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.025682 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4297b98b-9e10-4e72-a322-3e33984deb9e-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:22.025707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.025696 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngvwz\" (UniqueName: \"kubernetes.io/projected/4297b98b-9e10-4e72-a322-3e33984deb9e-kube-api-access-ngvwz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:22.503906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.503873 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" Oct 31 21:43:22.504365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.503881 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6" event={"ID":"4297b98b-9e10-4e72-a322-3e33984deb9e","Type":"ContainerDied","Data":"fce36a5ef2395d1a712c2ddea65f75e220bb389371555020742a63b3e34027d8"} Oct 31 21:43:22.504365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.504025 4294 scope.go:122] "RemoveContainer" containerID="6ecc70ced6671b2222b244b451be05401c6012afef6612170480d8324507757b" Oct 31 21:43:22.574884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.574852 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6"] Oct 31 21:43:22.586727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:22.586699 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-9qrn6"] Oct 31 21:43:23.985823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:23.985790 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" path="/var/lib/kubelet/pods/4297b98b-9e10-4e72-a322-3e33984deb9e/volumes" Oct 31 21:43:25.760158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.760126 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:43:25.789871 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.789846 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:43:25.881857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.881823 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-envoy\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.881891 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-credential-socket\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.881938 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-socket\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.881976 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-certs\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882002 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istiod-ca-cert\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882065 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-kube-api-access-8b9nn\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-kube-api-access-8b9nn\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882113 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-socket\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882192 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-certs\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882217 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-token\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-token\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882215 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-credential-socket" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.882256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882249 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-podinfo\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882333 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-kube-api-access-kjqwn\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-kube-api-access-kjqwn\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882396 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-token\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882450 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089877af-57d8-4b4a-9c4d-838f74d53ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089877af-57d8-4b4a-9c4d-838f74d53ae9-istiod-ca-cert\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882499 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-data\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882531 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-credential-socket\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882598 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-envoy\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882623 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-podinfo\") pod \"089877af-57d8-4b4a-9c4d-838f74d53ae9\" (UID: \"089877af-57d8-4b4a-9c4d-838f74d53ae9\") " Oct 31 21:43:25.882809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882685 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-data\") pod \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\" (UID: \"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f\") " Oct 31 21:43:25.882809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882732 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-socket" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.883463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.882950 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-certs" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.883463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883033 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089877af-57d8-4b4a-9c4d-838f74d53ae9-istiod-ca-cert" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:25.883463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883253 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istiod-ca-cert" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:25.883624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883554 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-credential-socket" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.883624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883561 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.883624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883585 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.883624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883601 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.883624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883617 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.883858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883630 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/089877af-57d8-4b4a-9c4d-838f74d53ae9-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.883858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.883842 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-data" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.884516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.884210 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-socket" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.884516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.884493 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-envoy" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.885086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.885062 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-data" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.885223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.885201 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-certs" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.886204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.886176 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-podinfo" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:43:25.886773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.886747 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-envoy" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:25.886920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.886901 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-token" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:25.887077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.887055 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-kube-api-access-8b9nn" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "kube-api-access-8b9nn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:25.888024 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.887997 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-token" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:25.888462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.888420 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-podinfo" pod "089877af-57d8-4b4a-9c4d-838f74d53ae9" (UID: "089877af-57d8-4b4a-9c4d-838f74d53ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:43:25.888462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.888422 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-kube-api-access-kjqwn" pod "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" (UID: "2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f"). InnerVolumeSpecName "kube-api-access-kjqwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:25.985395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985346 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985382 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985396 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985410 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985426 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b9nn\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-kube-api-access-8b9nn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985439 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985451 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985463 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjqwn\" (UniqueName: \"kubernetes.io/projected/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-kube-api-access-kjqwn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985475 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985489 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985501 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985513 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:25.985706 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:25.985525 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/089877af-57d8-4b4a-9c4d-838f74d53ae9-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:26.535150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.535103 4294 generic.go:356] "Generic (PLEG): container finished" podID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerID="30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b" exitCode=0 Oct 31 21:43:26.535349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.535160 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" Oct 31 21:43:26.535349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.535186 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" event={"ID":"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f","Type":"ContainerDied","Data":"30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b"} Oct 31 21:43:26.535349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.535256 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf" event={"ID":"2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f","Type":"ContainerDied","Data":"035f1858bba8537c765cb3db91a0920c68390fc7a193e8ed66dfd05e4c07b46f"} Oct 31 21:43:26.535349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.535288 4294 scope.go:122] "RemoveContainer" containerID="30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b" Oct 31 21:43:26.537205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.537184 4294 generic.go:356] "Generic (PLEG): container finished" podID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerID="e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a" exitCode=0 Oct 31 21:43:26.537320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.537230 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" event={"ID":"089877af-57d8-4b4a-9c4d-838f74d53ae9","Type":"ContainerDied","Data":"e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a"} Oct 31 21:43:26.537320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.537268 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" event={"ID":"089877af-57d8-4b4a-9c4d-838f74d53ae9","Type":"ContainerDied","Data":"bd27042a164be52c742d9c8ad831941a770ca796ed3aa03501649cea6160c0ba"} Oct 31 21:43:26.537320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.537284 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc" Oct 31 21:43:26.546720 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.546706 4294 scope.go:122] "RemoveContainer" containerID="30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b" Oct 31 21:43:26.547009 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:26.546993 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b\": container with ID starting with 30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b not found: ID does not exist" containerID="30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b" Oct 31 21:43:26.547046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.547018 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b"} err="failed to get container status \"30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b\": rpc error: code = NotFound desc = could not find container \"30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b\": container with ID starting with 30bdda55f65cfb737e79125be41d949f04c97a3696c640f258fcd267f50af34b not found: ID does not exist" Oct 31 21:43:26.547046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.547034 4294 scope.go:122] "RemoveContainer" containerID="e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a" Oct 31 21:43:26.555791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.555769 4294 scope.go:122] "RemoveContainer" containerID="e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a" Oct 31 21:43:26.556034 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:26.556014 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a\": container with ID starting with e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a not found: ID does not exist" containerID="e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a" Oct 31 21:43:26.556103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.556045 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a"} err="failed to get container status \"e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a\": rpc error: code = NotFound desc = could not find container \"e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a\": container with ID starting with e377f445313ca9b8396eb3bea0d3b2538030f12b26a5328057c9b5b1e0c40f6a not found: ID does not exist" Oct 31 21:43:26.605314 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.605273 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc"] Oct 31 21:43:26.617285 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.617254 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-l5ljc"] Oct 31 21:43:26.646790 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.646760 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf"] Oct 31 21:43:26.649387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:26.649365 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-sxhnf"] Oct 31 21:43:27.988687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:27.988649 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" path="/var/lib/kubelet/pods/089877af-57d8-4b4a-9c4d-838f74d53ae9/volumes" Oct 31 21:43:27.989485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:27.989463 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" path="/var/lib/kubelet/pods/2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f/volumes" Oct 31 21:43:28.052428 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:28.052386 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr"] Oct 31 21:43:28.052668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:28.052638 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" containerID="cri-o://6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e" gracePeriod=30 Oct 31 21:43:31.310696 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:31.310652 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:43:32.260601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.260565 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm"] Oct 31 21:43:32.261842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261812 4294 cpu_manager.go:414] "containerMap: removing container" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" Oct 31 21:43:32.261842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261843 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261869 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261877 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261890 4294 cpu_manager.go:414] "containerMap: removing container" podUID="069ea2b3-55cc-45b7-9a52-95897f5c59cc" containerName="maas-api" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261898 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069ea2b3-55cc-45b7-9a52-95897f5c59cc" containerName="maas-api" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261918 4294 cpu_manager.go:414] "containerMap: removing container" podUID="51997c34-e3a4-4981-8fab-6e4f0ccd66ce" containerName="revoke-keys" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261926 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51997c34-e3a4-4981-8fab-6e4f0ccd66ce" containerName="revoke-keys" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261938 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261945 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261984 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" Oct 31 21:43:32.262032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.261990 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" Oct 31 21:43:32.262342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.262140 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="089877af-57d8-4b4a-9c4d-838f74d53ae9" containerName="istio-proxy" Oct 31 21:43:32.262342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.262159 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a24e18f-9d3c-4e4b-8eaf-9703c80cfa6f" containerName="istio-proxy" Oct 31 21:43:32.262342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.262169 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="069ea2b3-55cc-45b7-9a52-95897f5c59cc" containerName="maas-api" Oct 31 21:43:32.262342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.262179 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="51997c34-e3a4-4981-8fab-6e4f0ccd66ce" containerName="revoke-keys" Oct 31 21:43:32.262342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.262191 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4297b98b-9e10-4e72-a322-3e33984deb9e" containerName="istio-proxy" Oct 31 21:43:32.262342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.262203 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b9fc8b4-6465-412a-84d8-3cef75fb8ffd" containerName="istio-proxy" Oct 31 21:43:32.267144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.267122 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.301280 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.301232 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm"] Oct 31 21:43:32.301468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.301395 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-09f39e-openshift-default-dockercfg-9wnsl\"" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390029 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istiod-ca-cert\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390083 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-certs\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390155 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-envoy\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390270 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-podinfo\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390470 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lfct\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-kube-api-access-4lfct\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390577 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-data\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390659 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-socket\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390695 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-credential-socket\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.392331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.390723 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-token\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.497777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.497733 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-data\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.498080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.498056 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-socket\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.498216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.498200 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-credential-socket\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.498431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.498410 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-token\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.498663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.498644 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istiod-ca-cert\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.498799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.498783 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-certs\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.498985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.498967 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-envoy\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.499157 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.499142 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-podinfo\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.499295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.499280 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lfct\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-kube-api-access-4lfct\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.500288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.500264 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-data\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.500845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.500814 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-certs\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.501075 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.501045 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istiod-ca-cert\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.501510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.501487 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-socket\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.502009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.501989 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-credential-socket\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.523994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.523905 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-envoy\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.524247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.524173 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-podinfo\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.536994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.536963 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lfct\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-kube-api-access-4lfct\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.537637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.537612 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-token\") pod \"e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.585447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.585167 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:32.974657 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.974627 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm"] Oct 31 21:43:32.975968 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:43:32.975939 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9df0bfc_50ed_49c8_98f2_d0803059d97d.slice/crio-ad80be5928b7ee8b096ca29d431cb692b65357ae6f0a1d63933b8edc1412e0f7 WatchSource:0}: Error finding container ad80be5928b7ee8b096ca29d431cb692b65357ae6f0a1d63933b8edc1412e0f7: Status 404 returned error can't find the container with id ad80be5928b7ee8b096ca29d431cb692b65357ae6f0a1d63933b8edc1412e0f7 Oct 31 21:43:32.981240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.981199 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:43:32.981350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.981294 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:43:32.981402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:32.981385 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:43:33.412712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.412682 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:43:33.544121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544087 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-socket\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544127 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-data\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544226 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-certs\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544269 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11df5d62-c1f6-4651-b068-b0bacee10d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11df5d62-c1f6-4651-b068-b0bacee10d4a-istiod-ca-cert\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544295 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-token\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544365 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-podinfo\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544414 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-kube-api-access-pjhmf\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-kube-api-access-pjhmf\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544479 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-socket" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:33.544741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544534 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-credential-socket\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544559 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-data" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:33.544741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544626 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-envoy\") pod \"11df5d62-c1f6-4651-b068-b0bacee10d4a\" (UID: \"11df5d62-c1f6-4651-b068-b0bacee10d4a\") " Oct 31 21:43:33.544893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544808 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-credential-socket" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:33.544893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544824 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11df5d62-c1f6-4651-b068-b0bacee10d4a-istiod-ca-cert" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:33.545005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.544979 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-certs" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:33.545345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.545323 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.545345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.545343 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.545510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.545352 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.545510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.545361 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.545510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.545369 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11df5d62-c1f6-4651-b068-b0bacee10d4a-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.546835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.546802 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-token" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:33.547102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.547079 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-kube-api-access-pjhmf" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "kube-api-access-pjhmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:33.547237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.547220 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-envoy" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:33.547784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.547762 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-podinfo" pod "11df5d62-c1f6-4651-b068-b0bacee10d4a" (UID: "11df5d62-c1f6-4651-b068-b0bacee10d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:43:33.586459 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.586426 4294 generic.go:356] "Generic (PLEG): container finished" podID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerID="6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e" exitCode=0 Oct 31 21:43:33.586596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.586474 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" Oct 31 21:43:33.586596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.586510 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" event={"ID":"11df5d62-c1f6-4651-b068-b0bacee10d4a","Type":"ContainerDied","Data":"6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e"} Oct 31 21:43:33.586596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.586567 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr" event={"ID":"11df5d62-c1f6-4651-b068-b0bacee10d4a","Type":"ContainerDied","Data":"d54896326abc687d18f5c8673346209c21181778bc679b68ece868f5e466dd00"} Oct 31 21:43:33.586699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.586599 4294 scope.go:122] "RemoveContainer" containerID="6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e" Oct 31 21:43:33.588666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.588640 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" event={"ID":"c9df0bfc-50ed-49c8-98f2-d0803059d97d","Type":"ContainerStarted","Data":"9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26"} Oct 31 21:43:33.588758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.588690 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" event={"ID":"c9df0bfc-50ed-49c8-98f2-d0803059d97d","Type":"ContainerStarted","Data":"ad80be5928b7ee8b096ca29d431cb692b65357ae6f0a1d63933b8edc1412e0f7"} Oct 31 21:43:33.593859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.593829 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx"] Oct 31 21:43:33.594402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.594384 4294 cpu_manager.go:414] "containerMap: removing container" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" Oct 31 21:43:33.594489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.594403 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" Oct 31 21:43:33.594489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.594433 4294 cpu_manager.go:414] "containerMap: removing container" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" Oct 31 21:43:33.594489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.594441 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" Oct 31 21:43:33.594634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.594622 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" Oct 31 21:43:33.598287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.598174 4294 scope.go:122] "RemoveContainer" containerID="f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18" Oct 31 21:43:33.599491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.599475 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.607810 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.607788 4294 scope.go:122] "RemoveContainer" containerID="6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e" Oct 31 21:43:33.608107 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:33.608086 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e\": container with ID starting with 6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e not found: ID does not exist" containerID="6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e" Oct 31 21:43:33.608197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.608114 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e"} err="failed to get container status \"6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e\": rpc error: code = NotFound desc = could not find container \"6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e\": container with ID starting with 6f665f1c1ee98520fc4bebc39bf8db7cd5b25ea5aa9bc5b5a538d8ec9b426b2e not found: ID does not exist" Oct 31 21:43:33.608197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.608131 4294 scope.go:122] "RemoveContainer" containerID="f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18" Oct 31 21:43:33.608470 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:33.608449 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18\": container with ID starting with f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18 not found: ID does not exist" containerID="f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18" Oct 31 21:43:33.608549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.608476 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18"} err="failed to get container status \"f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18\": rpc error: code = NotFound desc = could not find container \"f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18\": container with ID starting with f67b857b895943a0210fd2d45ee1b9a0f581a1a48b2546d5b2c2be934ee15c18 not found: ID does not exist" Oct 31 21:43:33.615699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.615678 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:43:33.616154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.615802 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:43:33.622176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.620331 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx"] Oct 31 21:43:33.646223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.646186 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb9lk\" (UniqueName: \"kubernetes.io/projected/2949b0be-fa39-4f9e-8499-0be56f6aac49-kube-api-access-zb9lk\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.646509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.646459 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.646594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.646569 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.646679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.646662 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.646754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.646683 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.646754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.646697 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjhmf\" (UniqueName: \"kubernetes.io/projected/11df5d62-c1f6-4651-b068-b0bacee10d4a-kube-api-access-pjhmf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.646754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.646710 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11df5d62-c1f6-4651-b068-b0bacee10d4a-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:33.685623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.685585 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr"] Oct 31 21:43:33.691739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.691714 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-fc0d65-openshift-default-84f596955-qwqmr"] Oct 31 21:43:33.765324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.747659 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.765324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.747781 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb9lk\" (UniqueName: \"kubernetes.io/projected/2949b0be-fa39-4f9e-8499-0be56f6aac49-kube-api-access-zb9lk\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.765324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.748363 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podStartSLOduration=1.7483468439999998 podStartE2EDuration="1.748346844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:33.747069677 +0000 UTC m=+1653.872176909" watchObservedRunningTime="2026-10-31 21:43:33.748346844 +0000 UTC m=+1653.873454075" Oct 31 21:43:33.765324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.756738 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.765324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.757588 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.775320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.766721 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.783712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.783652 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb9lk\" (UniqueName: \"kubernetes.io/projected/2949b0be-fa39-4f9e-8499-0be56f6aac49-kube-api-access-zb9lk\") pod \"maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.918850 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.918741 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:33.991344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:33.990470 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" path="/var/lib/kubelet/pods/11df5d62-c1f6-4651-b068-b0bacee10d4a/volumes" Oct 31 21:43:34.260364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:34.260333 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx"] Oct 31 21:43:34.261612 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:43:34.261584 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2949b0be_fa39_4f9e_8499_0be56f6aac49.slice/crio-643dee46c64adda452f518cc50ceaebeffac887726fd8a7c5508aa6b752fd9c7 WatchSource:0}: Error finding container 643dee46c64adda452f518cc50ceaebeffac887726fd8a7c5508aa6b752fd9c7: Status 404 returned error can't find the container with id 643dee46c64adda452f518cc50ceaebeffac887726fd8a7c5508aa6b752fd9c7 Oct 31 21:43:34.586211 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:34.586180 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:34.593268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:34.593247 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:34.597137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:34.597108 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" event={"ID":"2949b0be-fa39-4f9e-8499-0be56f6aac49","Type":"ContainerStarted","Data":"57ffc4427bc3591d8c5f8120a3bb20feab8db303f71c403b7cc6ef44115fda13"} Oct 31 21:43:34.597248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:34.597162 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" event={"ID":"2949b0be-fa39-4f9e-8499-0be56f6aac49","Type":"ContainerStarted","Data":"643dee46c64adda452f518cc50ceaebeffac887726fd8a7c5508aa6b752fd9c7"} Oct 31 21:43:34.598513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:34.598494 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:34.599408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:34.599389 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:43:35.620646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:35.620610 4294 generic.go:356] "Generic (PLEG): container finished" podID="2949b0be-fa39-4f9e-8499-0be56f6aac49" containerID="57ffc4427bc3591d8c5f8120a3bb20feab8db303f71c403b7cc6ef44115fda13" exitCode=0 Oct 31 21:43:35.621125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:35.620658 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" event={"ID":"2949b0be-fa39-4f9e-8499-0be56f6aac49","Type":"ContainerDied","Data":"57ffc4427bc3591d8c5f8120a3bb20feab8db303f71c403b7cc6ef44115fda13"} Oct 31 21:43:36.792234 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.792208 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:36.905999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.905903 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2949b0be-fa39-4f9e-8499-0be56f6aac49-kube-api-access-zb9lk\" (UniqueName: \"kubernetes.io/projected/2949b0be-fa39-4f9e-8499-0be56f6aac49-kube-api-access-zb9lk\") pod \"2949b0be-fa39-4f9e-8499-0be56f6aac49\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " Oct 31 21:43:36.905999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.905994 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-openshift-ca\") pod \"2949b0be-fa39-4f9e-8499-0be56f6aac49\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " Oct 31 21:43:36.906244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.906067 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-xks-ca\") pod \"2949b0be-fa39-4f9e-8499-0be56f6aac49\" (UID: \"2949b0be-fa39-4f9e-8499-0be56f6aac49\") " Oct 31 21:43:36.906460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.906425 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-openshift-ca" pod "2949b0be-fa39-4f9e-8499-0be56f6aac49" (UID: "2949b0be-fa39-4f9e-8499-0be56f6aac49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:36.906576 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.906554 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:36.908064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.908037 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-xks-ca" pod "2949b0be-fa39-4f9e-8499-0be56f6aac49" (UID: "2949b0be-fa39-4f9e-8499-0be56f6aac49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:43:36.908149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:36.908113 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2949b0be-fa39-4f9e-8499-0be56f6aac49-kube-api-access-zb9lk" pod "2949b0be-fa39-4f9e-8499-0be56f6aac49" (UID: "2949b0be-fa39-4f9e-8499-0be56f6aac49"). InnerVolumeSpecName "kube-api-access-zb9lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:37.008103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:37.008072 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb9lk\" (UniqueName: \"kubernetes.io/projected/2949b0be-fa39-4f9e-8499-0be56f6aac49-kube-api-access-zb9lk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:37.008103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:37.008099 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2949b0be-fa39-4f9e-8499-0be56f6aac49-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:37.641884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:37.641852 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" Oct 31 21:43:37.641884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:37.641867 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx" event={"ID":"2949b0be-fa39-4f9e-8499-0be56f6aac49","Type":"ContainerDied","Data":"643dee46c64adda452f518cc50ceaebeffac887726fd8a7c5508aa6b752fd9c7"} Oct 31 21:43:37.642102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:37.641917 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="643dee46c64adda452f518cc50ceaebeffac887726fd8a7c5508aa6b752fd9c7" Oct 31 21:43:38.233995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.233963 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq"] Oct 31 21:43:38.234424 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.234169 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" containerID="cri-o://4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776" gracePeriod=30 Oct 31 21:43:38.711417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.711332 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj"] Oct 31 21:43:38.711877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.711862 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2949b0be-fa39-4f9e-8499-0be56f6aac49" containerName="revoke-keys" Oct 31 21:43:38.711960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.711877 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2949b0be-fa39-4f9e-8499-0be56f6aac49" containerName="revoke-keys" Oct 31 21:43:38.712015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.711962 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2949b0be-fa39-4f9e-8499-0be56f6aac49" containerName="revoke-keys" Oct 31 21:43:38.712015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.711972 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="11df5d62-c1f6-4651-b068-b0bacee10d4a" containerName="istio-proxy" Oct 31 21:43:38.715464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.715439 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:38.753009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.752978 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-09f39e\"" Oct 31 21:43:38.753443 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.753421 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-09f39e-dockercfg-7x42p\"" Oct 31 21:43:38.758116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.758095 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj"] Oct 31 21:43:38.802702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.802667 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw"] Oct 31 21:43:38.811362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.810914 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:38.912365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.872353 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4srn\" (UniqueName: \"kubernetes.io/projected/a82c25f0-7237-4aa9-ad16-16b19c89a321-kube-api-access-m4srn\") pod \"payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:38.912365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.909495 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af7807-56f5-4bc3-9ce5-6e75b541ba05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:38.912365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.909567 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2tv\" (UniqueName: \"kubernetes.io/projected/57af7807-56f5-4bc3-9ce5-6e75b541ba05-kube-api-access-hs2tv\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:38.912365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.909699 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82c25f0-7237-4aa9-ad16-16b19c89a321-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:38.929957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:38.929916 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw"] Oct 31 21:43:39.011748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.011711 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4srn\" (UniqueName: \"kubernetes.io/projected/a82c25f0-7237-4aa9-ad16-16b19c89a321-kube-api-access-m4srn\") pod \"payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:39.012077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.011832 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af7807-56f5-4bc3-9ce5-6e75b541ba05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:39.012077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.011884 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2tv\" (UniqueName: \"kubernetes.io/projected/57af7807-56f5-4bc3-9ce5-6e75b541ba05-kube-api-access-hs2tv\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:39.012077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.011991 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82c25f0-7237-4aa9-ad16-16b19c89a321-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:39.012948 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.012922 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82c25f0-7237-4aa9-ad16-16b19c89a321-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:39.013255 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.013221 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af7807-56f5-4bc3-9ce5-6e75b541ba05-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:39.036656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.036628 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2tv\" (UniqueName: \"kubernetes.io/projected/57af7807-56f5-4bc3-9ce5-6e75b541ba05-kube-api-access-hs2tv\") pod \"payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:39.036656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.036644 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4srn\" (UniqueName: \"kubernetes.io/projected/a82c25f0-7237-4aa9-ad16-16b19c89a321-kube-api-access-m4srn\") pod \"payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:39.230393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.230360 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:39.332777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.332733 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:39.553030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.552951 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw"] Oct 31 21:43:39.557018 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:43:39.556991 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82c25f0_7237_4aa9_ad16_16b19c89a321.slice/crio-6947e145a104bab5b71b41379e312f2ddbc0f1b7ced20bea1fd72ced1ddacb13 WatchSource:0}: Error finding container 6947e145a104bab5b71b41379e312f2ddbc0f1b7ced20bea1fd72ced1ddacb13: Status 404 returned error can't find the container with id 6947e145a104bab5b71b41379e312f2ddbc0f1b7ced20bea1fd72ced1ddacb13 Oct 31 21:43:39.658256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.658223 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" event={"ID":"a82c25f0-7237-4aa9-ad16-16b19c89a321","Type":"ContainerStarted","Data":"f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6"} Oct 31 21:43:39.658404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.658282 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" event={"ID":"a82c25f0-7237-4aa9-ad16-16b19c89a321","Type":"ContainerStarted","Data":"6947e145a104bab5b71b41379e312f2ddbc0f1b7ced20bea1fd72ced1ddacb13"} Oct 31 21:43:39.658501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.658484 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:39.668969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:39.668943 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj"] Oct 31 21:43:40.665821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:40.665779 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" event={"ID":"57af7807-56f5-4bc3-9ce5-6e75b541ba05","Type":"ContainerStarted","Data":"914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead"} Oct 31 21:43:40.666293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:40.665843 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" event={"ID":"57af7807-56f5-4bc3-9ce5-6e75b541ba05","Type":"ContainerStarted","Data":"a829fe912b43ace31402513166cbda9c0eced4e361640c555db9d8a7a7f91dad"} Oct 31 21:43:40.666293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:40.666078 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:40.708554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:40.708506 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" podStartSLOduration=2.70849166 podStartE2EDuration="2.70849166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:39.718998763 +0000 UTC m=+1659.844105994" watchObservedRunningTime="2026-10-31 21:43:40.70849166 +0000 UTC m=+1660.833598921" Oct 31 21:43:40.709053 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:40.709033 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" podStartSLOduration=2.709027379 podStartE2EDuration="2.709027379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:43:40.706449294 +0000 UTC m=+1660.831556526" watchObservedRunningTime="2026-10-31 21:43:40.709027379 +0000 UTC m=+1660.834134646" Oct 31 21:43:42.802483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.802400 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m"] Oct 31 21:43:42.806520 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.806499 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.827634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.827607 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m"] Oct 31 21:43:42.833895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.833346 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:43:42.833895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.833392 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:43:42.868191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.868158 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.868432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.868410 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plb2v\" (UniqueName: \"kubernetes.io/projected/e8e3e01b-8777-4612-b986-c19c87a1df40-kube-api-access-plb2v\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.868607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.868590 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.970083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.970050 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plb2v\" (UniqueName: \"kubernetes.io/projected/e8e3e01b-8777-4612-b986-c19c87a1df40-kube-api-access-plb2v\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.970259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.970169 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.970259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.970222 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.970944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.970918 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.972560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.972540 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:42.994896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:42.994851 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plb2v\" (UniqueName: \"kubernetes.io/projected/e8e3e01b-8777-4612-b986-c19c87a1df40-kube-api-access-plb2v\") pod \"maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:43.120282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.120198 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:43.459298 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.459267 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m"] Oct 31 21:43:43.461253 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:43:43.461210 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e3e01b_8777_4612_b986_c19c87a1df40.slice/crio-376f522680fe9c06c2e12cd40e191d97adf9c85cb1c7cbb09a7b6dc82f6d22cc WatchSource:0}: Error finding container 376f522680fe9c06c2e12cd40e191d97adf9c85cb1c7cbb09a7b6dc82f6d22cc: Status 404 returned error can't find the container with id 376f522680fe9c06c2e12cd40e191d97adf9c85cb1c7cbb09a7b6dc82f6d22cc Oct 31 21:43:43.487592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.487569 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:43:43.580755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.580725 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-certs\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.580950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.580767 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-envoy\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.580950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.580816 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-kube-api-access-pvm8w\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-kube-api-access-pvm8w\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.580950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.580893 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0121bcf5-aee6-409b-982a-6f2939da08b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121bcf5-aee6-409b-982a-6f2939da08b6-istiod-ca-cert\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.581124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.580957 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-credential-socket\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.581124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581051 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-data\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.581124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581048 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-certs" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:43.581283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581174 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-token\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-token\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.581283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581220 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-socket\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.581283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581252 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-podinfo\") pod \"0121bcf5-aee6-409b-982a-6f2939da08b6\" (UID: \"0121bcf5-aee6-409b-982a-6f2939da08b6\") " Oct 31 21:43:43.581464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581335 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0121bcf5-aee6-409b-982a-6f2939da08b6-istiod-ca-cert" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:43.581464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581442 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-data" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:43.581591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581565 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-socket" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:43.581962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581937 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.582083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581967 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.582083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581983 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.582083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.581997 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0121bcf5-aee6-409b-982a-6f2939da08b6-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.582449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.582425 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-credential-socket" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:43.583248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.583225 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-envoy" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:43:43.583457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.583434 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-podinfo" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:43:43.583750 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.583722 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-kube-api-access-pvm8w" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "kube-api-access-pvm8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:43.584278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.584251 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-token" pod "0121bcf5-aee6-409b-982a-6f2939da08b6" (UID: "0121bcf5-aee6-409b-982a-6f2939da08b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:43.684139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.683999 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.684139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.684040 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvm8w\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-kube-api-access-pvm8w\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.684139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.684056 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0121bcf5-aee6-409b-982a-6f2939da08b6-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.684139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.684071 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.684529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.684208 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0121bcf5-aee6-409b-982a-6f2939da08b6-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:43.692721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.692693 4294 generic.go:356] "Generic (PLEG): container finished" podID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerID="4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776" exitCode=0 Oct 31 21:43:43.692811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.692725 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" event={"ID":"0121bcf5-aee6-409b-982a-6f2939da08b6","Type":"ContainerDied","Data":"4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776"} Oct 31 21:43:43.692811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.692747 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" Oct 31 21:43:43.692811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.692779 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq" event={"ID":"0121bcf5-aee6-409b-982a-6f2939da08b6","Type":"ContainerDied","Data":"455eb8f14b596fd7130594dc5eb8748b927bfcee692475b52c1bf4c3746b5730"} Oct 31 21:43:43.692811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.692800 4294 scope.go:122] "RemoveContainer" containerID="4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776" Oct 31 21:43:43.694360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.694341 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" event={"ID":"e8e3e01b-8777-4612-b986-c19c87a1df40","Type":"ContainerStarted","Data":"376f522680fe9c06c2e12cd40e191d97adf9c85cb1c7cbb09a7b6dc82f6d22cc"} Oct 31 21:43:43.702666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.702647 4294 scope.go:122] "RemoveContainer" containerID="4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776" Oct 31 21:43:43.702921 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:43:43.702904 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776\": container with ID starting with 4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776 not found: ID does not exist" containerID="4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776" Oct 31 21:43:43.702961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.702927 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776"} err="failed to get container status \"4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776\": rpc error: code = NotFound desc = could not find container \"4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776\": container with ID starting with 4daaaef9fe872ae767ef46d1eb0f5b37976812be82d8f18d51e88184d6fa8776 not found: ID does not exist" Oct 31 21:43:43.757239 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.757206 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq"] Oct 31 21:43:43.785475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.785447 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-bncsq"] Oct 31 21:43:43.986607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:43.986576 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" path="/var/lib/kubelet/pods/0121bcf5-aee6-409b-982a-6f2939da08b6/volumes" Oct 31 21:43:44.703475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:44.703368 4294 generic.go:356] "Generic (PLEG): container finished" podID="e8e3e01b-8777-4612-b986-c19c87a1df40" containerID="0c1c6c06183da4b53defa798d3c06d96b04988754c7d389a961f7d6229230b71" exitCode=0 Oct 31 21:43:44.703475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:44.703415 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" event={"ID":"e8e3e01b-8777-4612-b986-c19c87a1df40","Type":"ContainerDied","Data":"0c1c6c06183da4b53defa798d3c06d96b04988754c7d389a961f7d6229230b71"} Oct 31 21:43:45.867377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:45.867350 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:46.012978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.012950 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e3e01b-8777-4612-b986-c19c87a1df40-kube-api-access-plb2v\" (UniqueName: \"kubernetes.io/projected/e8e3e01b-8777-4612-b986-c19c87a1df40-kube-api-access-plb2v\") pod \"e8e3e01b-8777-4612-b986-c19c87a1df40\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " Oct 31 21:43:46.013148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.013000 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-openshift-ca\") pod \"e8e3e01b-8777-4612-b986-c19c87a1df40\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " Oct 31 21:43:46.013148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.013023 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-xks-ca\") pod \"e8e3e01b-8777-4612-b986-c19c87a1df40\" (UID: \"e8e3e01b-8777-4612-b986-c19c87a1df40\") " Oct 31 21:43:46.013470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.013444 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-openshift-ca" pod "e8e3e01b-8777-4612-b986-c19c87a1df40" (UID: "e8e3e01b-8777-4612-b986-c19c87a1df40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:43:46.015045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.015022 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e3e01b-8777-4612-b986-c19c87a1df40-kube-api-access-plb2v" pod "e8e3e01b-8777-4612-b986-c19c87a1df40" (UID: "e8e3e01b-8777-4612-b986-c19c87a1df40"). InnerVolumeSpecName "kube-api-access-plb2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:43:46.015156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.015029 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-xks-ca" pod "e8e3e01b-8777-4612-b986-c19c87a1df40" (UID: "e8e3e01b-8777-4612-b986-c19c87a1df40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:43:46.114842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.114807 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plb2v\" (UniqueName: \"kubernetes.io/projected/e8e3e01b-8777-4612-b986-c19c87a1df40-kube-api-access-plb2v\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:46.114842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.114838 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:46.114842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.114848 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e3e01b-8777-4612-b986-c19c87a1df40-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:43:46.719132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.719095 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" Oct 31 21:43:46.719346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.719127 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m" event={"ID":"e8e3e01b-8777-4612-b986-c19c87a1df40","Type":"ContainerDied","Data":"376f522680fe9c06c2e12cd40e191d97adf9c85cb1c7cbb09a7b6dc82f6d22cc"} Oct 31 21:43:46.719346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:46.719194 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="376f522680fe9c06c2e12cd40e191d97adf9c85cb1c7cbb09a7b6dc82f6d22cc" Oct 31 21:43:50.669886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:50.669853 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:43:51.684342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:51.684279 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:43:55.144913 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:55.144875 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m"] Oct 31 21:43:55.149609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:55.149585 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-6d6576-p2z9m"] Oct 31 21:43:55.987450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:55.987413 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e3e01b-8777-4612-b986-c19c87a1df40" path="/var/lib/kubelet/pods/e8e3e01b-8777-4612-b986-c19c87a1df40/volumes" Oct 31 21:43:59.377247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:59.377212 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx"] Oct 31 21:43:59.392954 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:59.392912 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d75c94-4bwjx"] Oct 31 21:43:59.985337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:43:59.985285 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2949b0be-fa39-4f9e-8499-0be56f6aac49" path="/var/lib/kubelet/pods/2949b0be-fa39-4f9e-8499-0be56f6aac49/volumes" Oct 31 21:44:01.310878 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:01.310845 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:44:07.621588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:07.621553 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28"] Oct 31 21:44:07.625325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:07.622185 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" containerID="cri-o://3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a" gracePeriod=30 Oct 31 21:44:08.454864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.454829 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv"] Oct 31 21:44:08.455404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.455389 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e8e3e01b-8777-4612-b986-c19c87a1df40" containerName="revoke-keys" Oct 31 21:44:08.455473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.455405 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e3e01b-8777-4612-b986-c19c87a1df40" containerName="revoke-keys" Oct 31 21:44:08.455473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.455417 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" Oct 31 21:44:08.455473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.455422 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" Oct 31 21:44:08.455565 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.455534 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e3e01b-8777-4612-b986-c19c87a1df40" containerName="revoke-keys" Oct 31 21:44:08.455565 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.455543 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0121bcf5-aee6-409b-982a-6f2939da08b6" containerName="istio-proxy" Oct 31 21:44:08.459057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.458970 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.477453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.477430 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv"] Oct 31 21:44:08.506598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.506571 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:44:08.506767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.506624 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:44:08.591434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.591396 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.591627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.591576 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qff5s\" (UniqueName: \"kubernetes.io/projected/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-kube-api-access-qff5s\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.591696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.591639 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.693273 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.693243 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.693638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.693422 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qff5s\" (UniqueName: \"kubernetes.io/projected/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-kube-api-access-qff5s\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.693638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.693492 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.693960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.693932 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.695593 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.695570 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.716390 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.716336 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qff5s\" (UniqueName: \"kubernetes.io/projected/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-kube-api-access-qff5s\") pod \"maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:08.780119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:08.780078 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:09.093079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:09.093051 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv"] Oct 31 21:44:09.094434 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:44:09.094405 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94311211_8cf2_43ef_8d5b_5dfa4e6661d6.slice/crio-9bc42136c6eb21c23cfafb1e548539447dc188c17a061ec4ca7e65e860818fe6 WatchSource:0}: Error finding container 9bc42136c6eb21c23cfafb1e548539447dc188c17a061ec4ca7e65e860818fe6: Status 404 returned error can't find the container with id 9bc42136c6eb21c23cfafb1e548539447dc188c17a061ec4ca7e65e860818fe6 Oct 31 21:44:09.891331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:09.890566 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" event={"ID":"94311211-8cf2-43ef-8d5b-5dfa4e6661d6","Type":"ContainerStarted","Data":"e589d74323780e7ab03328d603d6e2f4668859a5badad5a7a6621b082a7b85d5"} Oct 31 21:44:09.891331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:09.890911 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" event={"ID":"94311211-8cf2-43ef-8d5b-5dfa4e6661d6","Type":"ContainerStarted","Data":"9bc42136c6eb21c23cfafb1e548539447dc188c17a061ec4ca7e65e860818fe6"} Oct 31 21:44:09.934563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:09.934513 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" podStartSLOduration=1.9344988760000001 podStartE2EDuration="1.934498876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:09.932160937 +0000 UTC m=+1690.057268168" watchObservedRunningTime="2026-10-31 21:44:09.934498876 +0000 UTC m=+1690.059606106" Oct 31 21:44:10.906952 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:10.906906 4294 generic.go:356] "Generic (PLEG): container finished" podID="94311211-8cf2-43ef-8d5b-5dfa4e6661d6" containerID="e589d74323780e7ab03328d603d6e2f4668859a5badad5a7a6621b082a7b85d5" exitCode=0 Oct 31 21:44:10.907499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:10.907051 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" event={"ID":"94311211-8cf2-43ef-8d5b-5dfa4e6661d6","Type":"ContainerDied","Data":"e589d74323780e7ab03328d603d6e2f4668859a5badad5a7a6621b082a7b85d5"} Oct 31 21:44:11.437632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.437597 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm"] Oct 31 21:44:11.441696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.441678 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.460737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.460707 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm"] Oct 31 21:44:11.534481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.534447 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.534652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.534543 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.534865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.534824 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqps\" (UniqueName: \"kubernetes.io/projected/bafd9e79-0153-473e-b0ad-0bf8771f370b-kube-api-access-4nqps\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.636586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.636552 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqps\" (UniqueName: \"kubernetes.io/projected/bafd9e79-0153-473e-b0ad-0bf8771f370b-kube-api-access-4nqps\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.636777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.636611 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.636777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.636700 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.637343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.637319 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.639179 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.639156 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.660409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.660382 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqps\" (UniqueName: \"kubernetes.io/projected/bafd9e79-0153-473e-b0ad-0bf8771f370b-kube-api-access-4nqps\") pod \"maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:11.754962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:11.754930 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:12.059660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.059634 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:12.066337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.066315 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm"] Oct 31 21:44:12.067439 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:44:12.067417 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafd9e79_0153_473e_b0ad_0bf8771f370b.slice/crio-244f3a1ef22987f99274cfe0129c2aed12b6ac25b5b4592eee0cab2a06ec5611 WatchSource:0}: Error finding container 244f3a1ef22987f99274cfe0129c2aed12b6ac25b5b4592eee0cab2a06ec5611: Status 404 returned error can't find the container with id 244f3a1ef22987f99274cfe0129c2aed12b6ac25b5b4592eee0cab2a06ec5611 Oct 31 21:44:12.145443 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.145410 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-xks-ca\") pod \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " Oct 31 21:44:12.145637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.145471 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-kube-api-access-qff5s\" (UniqueName: \"kubernetes.io/projected/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-kube-api-access-qff5s\") pod \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " Oct 31 21:44:12.145637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.145623 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-openshift-ca\") pod \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\" (UID: \"94311211-8cf2-43ef-8d5b-5dfa4e6661d6\") " Oct 31 21:44:12.146132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.146097 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-openshift-ca" pod "94311211-8cf2-43ef-8d5b-5dfa4e6661d6" (UID: "94311211-8cf2-43ef-8d5b-5dfa4e6661d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:12.147727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.147702 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-xks-ca" pod "94311211-8cf2-43ef-8d5b-5dfa4e6661d6" (UID: "94311211-8cf2-43ef-8d5b-5dfa4e6661d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:12.147829 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.147798 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-kube-api-access-qff5s" pod "94311211-8cf2-43ef-8d5b-5dfa4e6661d6" (UID: "94311211-8cf2-43ef-8d5b-5dfa4e6661d6"). InnerVolumeSpecName "kube-api-access-qff5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:12.247684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.247652 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.247684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.247686 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qff5s\" (UniqueName: \"kubernetes.io/projected/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-kube-api-access-qff5s\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.247866 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.247700 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94311211-8cf2-43ef-8d5b-5dfa4e6661d6-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.914043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.914018 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:44:12.920842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.920814 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" event={"ID":"bafd9e79-0153-473e-b0ad-0bf8771f370b","Type":"ContainerStarted","Data":"4ab0a63056f98dfcaaa4fdcbfb947c2546bb126f3723576915e98927d05492da"} Oct 31 21:44:12.920946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.920867 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" event={"ID":"bafd9e79-0153-473e-b0ad-0bf8771f370b","Type":"ContainerStarted","Data":"244f3a1ef22987f99274cfe0129c2aed12b6ac25b5b4592eee0cab2a06ec5611"} Oct 31 21:44:12.923460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.923433 4294 generic.go:356] "Generic (PLEG): container finished" podID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerID="3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a" exitCode=0 Oct 31 21:44:12.923616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.923484 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" Oct 31 21:44:12.923616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.923515 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" event={"ID":"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0","Type":"ContainerDied","Data":"3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a"} Oct 31 21:44:12.923616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.923583 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28" event={"ID":"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0","Type":"ContainerDied","Data":"75181f1369464e4a0b81862f8c26741dcc679356b1df750818a37216fe091a7c"} Oct 31 21:44:12.923786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.923617 4294 scope.go:122] "RemoveContainer" containerID="3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a" Oct 31 21:44:12.925401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.925367 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" event={"ID":"94311211-8cf2-43ef-8d5b-5dfa4e6661d6","Type":"ContainerDied","Data":"9bc42136c6eb21c23cfafb1e548539447dc188c17a061ec4ca7e65e860818fe6"} Oct 31 21:44:12.925506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.925407 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc42136c6eb21c23cfafb1e548539447dc188c17a061ec4ca7e65e860818fe6" Oct 31 21:44:12.925506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.925379 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv" Oct 31 21:44:12.934010 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.933992 4294 scope.go:122] "RemoveContainer" containerID="b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8" Oct 31 21:44:12.943722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.943703 4294 scope.go:122] "RemoveContainer" containerID="3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a" Oct 31 21:44:12.943990 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:12.943967 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a\": container with ID starting with 3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a not found: ID does not exist" containerID="3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a" Oct 31 21:44:12.944089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.943993 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a"} err="failed to get container status \"3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a\": rpc error: code = NotFound desc = could not find container \"3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a\": container with ID starting with 3372f989d697415df00a1b7a87a354069834c89a3a3097a55974493e05b1499a not found: ID does not exist" Oct 31 21:44:12.944089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.944009 4294 scope.go:122] "RemoveContainer" containerID="b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8" Oct 31 21:44:12.944225 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:12.944210 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8\": container with ID starting with b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8 not found: ID does not exist" containerID="b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8" Oct 31 21:44:12.944258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.944229 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8"} err="failed to get container status \"b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8\": rpc error: code = NotFound desc = could not find container \"b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8\": container with ID starting with b28d5813cc3dafe58c080990d5e3581f8960bfc3ef4f839f063ea6296aa0acc8 not found: ID does not exist" Oct 31 21:44:12.960067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960029 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istiod-ca-cert\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960131 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-certs\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960160 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-socket\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960234 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-credential-socket\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960319 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-envoy\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960366 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-podinfo\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960387 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-kube-api-access-kjbfd\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-kube-api-access-kjbfd\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960391 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istiod-ca-cert" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:12.960499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960438 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-data\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960449 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-certs" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:12.960499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960478 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-token\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-token\") pod \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\" (UID: \"2ef5b225-efbc-4f97-b9b7-300c9c97e9d0\") " Oct 31 21:44:12.960681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960485 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-socket" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:12.960681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960571 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-credential-socket" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:12.960799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.960766 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-data" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:12.961050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.961029 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.961176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.961051 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.961176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.961065 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.961176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.961077 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.961176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.961089 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:12.962685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.962660 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-envoy" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:12.962878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.962855 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-podinfo" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:44:12.962958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.962886 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-token" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:12.962958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:12.962930 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-kube-api-access-kjbfd" pod "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" (UID: "2ef5b225-efbc-4f97-b9b7-300c9c97e9d0"). InnerVolumeSpecName "kube-api-access-kjbfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:13.020801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.020753 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" podStartSLOduration=2.020739466 podStartE2EDuration="2.020739466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:13.0194387 +0000 UTC m=+1693.144545930" watchObservedRunningTime="2026-10-31 21:44:13.020739466 +0000 UTC m=+1693.145846697" Oct 31 21:44:13.062110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.062079 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:13.062110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.062105 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:13.062110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.062115 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjbfd\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-kube-api-access-kjbfd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:13.062525 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.062124 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:13.270210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.270182 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28"] Oct 31 21:44:13.272932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.272898 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d75c94-openshift-default-68bc89997-ksn28"] Oct 31 21:44:13.934265 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.934233 4294 generic.go:356] "Generic (PLEG): container finished" podID="bafd9e79-0153-473e-b0ad-0bf8771f370b" containerID="4ab0a63056f98dfcaaa4fdcbfb947c2546bb126f3723576915e98927d05492da" exitCode=0 Oct 31 21:44:13.934265 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.934265 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" event={"ID":"bafd9e79-0153-473e-b0ad-0bf8771f370b","Type":"ContainerDied","Data":"4ab0a63056f98dfcaaa4fdcbfb947c2546bb126f3723576915e98927d05492da"} Oct 31 21:44:13.995347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:13.995290 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" path="/var/lib/kubelet/pods/2ef5b225-efbc-4f97-b9b7-300c9c97e9d0/volumes" Oct 31 21:44:15.152455 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.152428 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:15.194808 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.194776 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafd9e79-0153-473e-b0ad-0bf8771f370b-kube-api-access-4nqps\" (UniqueName: \"kubernetes.io/projected/bafd9e79-0153-473e-b0ad-0bf8771f370b-kube-api-access-4nqps\") pod \"bafd9e79-0153-473e-b0ad-0bf8771f370b\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " Oct 31 21:44:15.195050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.195025 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-xks-ca\") pod \"bafd9e79-0153-473e-b0ad-0bf8771f370b\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " Oct 31 21:44:15.195409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.195376 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-openshift-ca\") pod \"bafd9e79-0153-473e-b0ad-0bf8771f370b\" (UID: \"bafd9e79-0153-473e-b0ad-0bf8771f370b\") " Oct 31 21:44:15.195950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.195907 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-openshift-ca" pod "bafd9e79-0153-473e-b0ad-0bf8771f370b" (UID: "bafd9e79-0153-473e-b0ad-0bf8771f370b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:15.197870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.197848 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafd9e79-0153-473e-b0ad-0bf8771f370b-kube-api-access-4nqps" pod "bafd9e79-0153-473e-b0ad-0bf8771f370b" (UID: "bafd9e79-0153-473e-b0ad-0bf8771f370b"). InnerVolumeSpecName "kube-api-access-4nqps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:15.197972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.197888 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-xks-ca" pod "bafd9e79-0153-473e-b0ad-0bf8771f370b" (UID: "bafd9e79-0153-473e-b0ad-0bf8771f370b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:15.297683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.297602 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:15.297683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.297634 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqps\" (UniqueName: \"kubernetes.io/projected/bafd9e79-0153-473e-b0ad-0bf8771f370b-kube-api-access-4nqps\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:15.297683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.297649 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bafd9e79-0153-473e-b0ad-0bf8771f370b-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:15.988124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.988096 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" Oct 31 21:44:15.988276 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.988135 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm" event={"ID":"bafd9e79-0153-473e-b0ad-0bf8771f370b","Type":"ContainerDied","Data":"244f3a1ef22987f99274cfe0129c2aed12b6ac25b5b4592eee0cab2a06ec5611"} Oct 31 21:44:15.988276 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:15.988183 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="244f3a1ef22987f99274cfe0129c2aed12b6ac25b5b4592eee0cab2a06ec5611" Oct 31 21:44:21.466152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:21.466112 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77"] Oct 31 21:44:21.466562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:21.466369 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" containerID="cri-o://521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa" gracePeriod=30 Oct 31 21:44:24.701538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:24.701496 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw"] Oct 31 21:44:24.702030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:24.701797 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" containerID="cri-o://4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952" gracePeriod=30 Oct 31 21:44:26.756963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.756936 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:44:26.888277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888241 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istiod-ca-cert\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888314 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-certs\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888446 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-kube-api-access-s4mmb\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-kube-api-access-s4mmb\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888585 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888483 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-envoy\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888585 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888539 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-podinfo\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888582 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-socket\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888617 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-certs" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:26.888694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888631 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-token\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888655 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istiod-ca-cert" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:26.888879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888742 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-credential-socket\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888824 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-data\") pod \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\" (UID: \"cbc6f25d-3819-456d-b152-a2fe2b4a7e11\") " Oct 31 21:44:26.888980 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.888956 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-socket" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:26.889133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.889105 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-credential-socket" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:26.889414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.889248 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-data" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:26.889693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.889669 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.889834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.889812 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.889834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.889836 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.890004 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.889849 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.890004 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.889864 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.891117 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.891089 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-podinfo" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:44:26.891338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.891296 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-kube-api-access-s4mmb" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "kube-api-access-s4mmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:26.891411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.891361 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-envoy" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:26.891637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.891618 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-token" pod "cbc6f25d-3819-456d-b152-a2fe2b4a7e11" (UID: "cbc6f25d-3819-456d-b152-a2fe2b4a7e11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:26.991054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.991019 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4mmb\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-kube-api-access-s4mmb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.991054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.991050 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.991264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.991064 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:26.991264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:26.991075 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbc6f25d-3819-456d-b152-a2fe2b4a7e11-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:27.063270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.063234 4294 generic.go:356] "Generic (PLEG): container finished" podID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerID="521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa" exitCode=0 Oct 31 21:44:27.063449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.063289 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" Oct 31 21:44:27.063449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.063350 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" event={"ID":"cbc6f25d-3819-456d-b152-a2fe2b4a7e11","Type":"ContainerDied","Data":"521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa"} Oct 31 21:44:27.063449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.063424 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77" event={"ID":"cbc6f25d-3819-456d-b152-a2fe2b4a7e11","Type":"ContainerDied","Data":"65fc50399a161e54b53fab23950b1137d1cadebcfe72ca4c356b76bb068ac348"} Oct 31 21:44:27.063559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.063454 4294 scope.go:122] "RemoveContainer" containerID="521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa" Oct 31 21:44:27.075006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.074983 4294 scope.go:122] "RemoveContainer" containerID="521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa" Oct 31 21:44:27.075322 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:27.075285 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa\": container with ID starting with 521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa not found: ID does not exist" containerID="521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa" Oct 31 21:44:27.075399 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.075329 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa"} err="failed to get container status \"521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa\": rpc error: code = NotFound desc = could not find container \"521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa\": container with ID starting with 521acb92409e581dee4b53a884cd252cfb678435cf2f732fd823b95111074caa not found: ID does not exist" Oct 31 21:44:27.128844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.125504 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77"] Oct 31 21:44:27.130985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.130951 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-94s77"] Oct 31 21:44:27.984928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:27.984895 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" path="/var/lib/kubelet/pods/cbc6f25d-3819-456d-b152-a2fe2b4a7e11/volumes" Oct 31 21:44:28.745818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.745782 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5"] Oct 31 21:44:28.746697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746667 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" Oct 31 21:44:28.746697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746695 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" Oct 31 21:44:28.746891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746722 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" Oct 31 21:44:28.746891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746731 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" Oct 31 21:44:28.746891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746751 4294 cpu_manager.go:414] "containerMap: removing container" podUID="94311211-8cf2-43ef-8d5b-5dfa4e6661d6" containerName="revoke-keys" Oct 31 21:44:28.746891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746759 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94311211-8cf2-43ef-8d5b-5dfa4e6661d6" containerName="revoke-keys" Oct 31 21:44:28.746891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746782 4294 cpu_manager.go:414] "containerMap: removing container" podUID="bafd9e79-0153-473e-b0ad-0bf8771f370b" containerName="revoke-keys" Oct 31 21:44:28.746891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746789 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafd9e79-0153-473e-b0ad-0bf8771f370b" containerName="revoke-keys" Oct 31 21:44:28.747218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746955 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" Oct 31 21:44:28.747218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746971 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbc6f25d-3819-456d-b152-a2fe2b4a7e11" containerName="istio-proxy" Oct 31 21:44:28.747218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746985 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="94311211-8cf2-43ef-8d5b-5dfa4e6661d6" containerName="revoke-keys" Oct 31 21:44:28.747218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.746995 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" Oct 31 21:44:28.747218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.747002 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafd9e79-0153-473e-b0ad-0bf8771f370b" containerName="revoke-keys" Oct 31 21:44:28.754730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.754709 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:28.770629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.770600 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5"] Oct 31 21:44:28.779778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.779750 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-80542b\"/\"openshift-service-ca.crt\"" Oct 31 21:44:28.779923 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.779812 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-80542b\"/\"kube-root-ca.crt\"" Oct 31 21:44:28.783449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.783425 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-80542b\"/\"default-dockercfg-tx5fd\"" Oct 31 21:44:28.783660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.783640 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-80542b\"/\"praxis-route-80542b-kserve-self-signed-certs\"" Oct 31 21:44:28.921936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.921889 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-model-cache\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:28.921936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.921944 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tmp-dir\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:28.922204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.922089 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh2wt\" (UniqueName: \"kubernetes.io/projected/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-kube-api-access-xh2wt\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:28.922204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.922168 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-home\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:28.922369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.922349 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-dshm\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:28.922428 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:28.922407 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tls-certs\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024216 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tls-certs\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024293 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-model-cache\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024354 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tmp-dir\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024387 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh2wt\" (UniqueName: \"kubernetes.io/projected/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-kube-api-access-xh2wt\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024425 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-home\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024554 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-dshm\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024709 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tmp-dir\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.024862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024726 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-model-cache\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.025167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.024954 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-home\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.026715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.026686 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-dshm\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.026900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.026880 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tls-certs\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.056924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.056885 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh2wt\" (UniqueName: \"kubernetes.io/projected/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-kube-api-access-xh2wt\") pod \"praxis-route-80542b-kserve-78857d84b7-mmrg5\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.092321 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.092269 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:29.118082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.118017 4294 generic.go:356] "Generic (PLEG): container finished" podID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerID="9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26" exitCode=0 Oct 31 21:44:29.118082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.118066 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" event={"ID":"c9df0bfc-50ed-49c8-98f2-d0803059d97d","Type":"ContainerDied","Data":"9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26"} Oct 31 21:44:29.118758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.118736 4294 scope.go:122] "RemoveContainer" containerID="9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26" Oct 31 21:44:29.121279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.120714 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:44:29.121279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.120781 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:44:29.121279 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.120821 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:44:29.506342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.506244 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5"] Oct 31 21:44:29.509782 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:44:29.509744 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29f0c6c5_d180_4725_a7e9_0727e74f4ecd.slice/crio-86c47893fd2afd075f7ce11ee7681e6c5abbb03347bfc8178bfa191852c50b56 WatchSource:0}: Error finding container 86c47893fd2afd075f7ce11ee7681e6c5abbb03347bfc8178bfa191852c50b56: Status 404 returned error can't find the container with id 86c47893fd2afd075f7ce11ee7681e6c5abbb03347bfc8178bfa191852c50b56 Oct 31 21:44:29.523087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.523049 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv"] Oct 31 21:44:29.527436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:29.527413 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-bb9990-j7qlv"] Oct 31 21:44:30.011651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.011615 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94311211-8cf2-43ef-8d5b-5dfa4e6661d6" path="/var/lib/kubelet/pods/94311211-8cf2-43ef-8d5b-5dfa4e6661d6/volumes" Oct 31 21:44:30.031682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.031660 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:44:30.140689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.140590 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" event={"ID":"c9df0bfc-50ed-49c8-98f2-d0803059d97d","Type":"ContainerStarted","Data":"8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911"} Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.142833 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" event={"ID":"29f0c6c5-d180-4725-a7e9-0727e74f4ecd","Type":"ContainerStarted","Data":"2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75"} Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.142876 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" event={"ID":"29f0c6c5-d180-4725-a7e9-0727e74f4ecd","Type":"ContainerStarted","Data":"86c47893fd2afd075f7ce11ee7681e6c5abbb03347bfc8178bfa191852c50b56"} Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.144483 4294 generic.go:356] "Generic (PLEG): container finished" podID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerID="4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952" exitCode=0 Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.144536 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" event={"ID":"93ba5bd9-d03d-4399-a8b7-173694f0a66f","Type":"ContainerDied","Data":"4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952"} Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.144572 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" event={"ID":"93ba5bd9-d03d-4399-a8b7-173694f0a66f","Type":"ContainerDied","Data":"1b832deabdcd345cf2d4404bf741537c4e88f41f598aad0941f7fcd58d9ac776"} Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.144598 4294 scope.go:122] "RemoveContainer" containerID="4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.144735 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145623 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istiod-ca-cert\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145697 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-podinfo\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145728 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-data\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145760 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-certs\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145787 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-token\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-token\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145842 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-credential-socket\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145874 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-socket\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.145952 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-kube-api-access-kh7hx\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-kube-api-access-kh7hx\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.146017 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-envoy\") pod \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\" (UID: \"93ba5bd9-d03d-4399-a8b7-173694f0a66f\") " Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.146043 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istiod-ca-cert" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.146399 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-data" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.146534 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.146552 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.147392 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-credential-socket" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.147423 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-certs" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.147447 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-socket" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:30.150739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.149214 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-podinfo" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:44:30.152449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.151824 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-kube-api-access-kh7hx" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "kube-api-access-kh7hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:30.154352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.152872 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-envoy" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:30.154352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.153882 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-token" pod "93ba5bd9-d03d-4399-a8b7-173694f0a66f" (UID: "93ba5bd9-d03d-4399-a8b7-173694f0a66f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:30.158729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.158708 4294 scope.go:122] "RemoveContainer" containerID="4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952" Oct 31 21:44:30.159037 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:30.159018 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952\": container with ID starting with 4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952 not found: ID does not exist" containerID="4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952" Oct 31 21:44:30.159115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.159044 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952"} err="failed to get container status \"4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952\": rpc error: code = NotFound desc = could not find container \"4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952\": container with ID starting with 4bc0e726923ffc1dc955a1e822d370e1e18b2cf999b64cb6bc997662c99db952 not found: ID does not exist" Oct 31 21:44:30.247910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.247875 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.247910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.247909 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.248116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.247922 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.248116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.247936 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.248116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.247948 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.248116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.247961 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93ba5bd9-d03d-4399-a8b7-173694f0a66f-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.248116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.247975 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh7hx\" (UniqueName: \"kubernetes.io/projected/93ba5bd9-d03d-4399-a8b7-173694f0a66f-kube-api-access-kh7hx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:30.249920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.249872 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" podStartSLOduration=2.249858737 podStartE2EDuration="2.249858737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:30.247251555 +0000 UTC m=+1710.372358785" watchObservedRunningTime="2026-10-31 21:44:30.249858737 +0000 UTC m=+1710.374965967" Oct 31 21:44:30.498336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.491751 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw"] Oct 31 21:44:30.508646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.506041 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-4kglw"] Oct 31 21:44:30.596327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.595365 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:44:30.596752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.596714 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:44:30.596879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:30.596785 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 21:44:31.095326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:31.094702 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:31.325724 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:31.325687 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:44:31.590332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:31.590056 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:44:31.590332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:31.590136 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 21:44:32.003603 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:32.003562 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" path="/var/lib/kubelet/pods/93ba5bd9-d03d-4399-a8b7-173694f0a66f/volumes" Oct 31 21:44:32.519207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:32.519165 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj"] Oct 31 21:44:32.519689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:32.519388 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" podUID="2aefdb23-64c9-4043-8ba2-fee8d84a0860" containerName="discovery" containerID="cri-o://c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645" gracePeriod=30 Oct 31 21:44:32.597328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:32.596550 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:44:32.597328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:32.596637 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 21:44:32.832232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:32.832209 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:44:33.002832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.002799 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-token\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-token\") pod \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " Oct 31 21:44:33.002832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.002845 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-ca-configmap\") pod \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " Oct 31 21:44:33.003146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.002934 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-dns-cert\") pod \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " Oct 31 21:44:33.003146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.003001 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-kubeconfig\") pod \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " Oct 31 21:44:33.003146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.003074 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-cacerts\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-cacerts\") pod \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " Oct 31 21:44:33.003331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.003253 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aefdb23-64c9-4043-8ba2-fee8d84a0860-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aefdb23-64c9-4043-8ba2-fee8d84a0860-local-certs\") pod \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " Oct 31 21:44:33.003331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.003292 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-kube-api-access-jqpm9\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-kube-api-access-jqpm9\") pod \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\" (UID: \"2aefdb23-64c9-4043-8ba2-fee8d84a0860\") " Oct 31 21:44:33.003993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.003943 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-ca-configmap" pod "2aefdb23-64c9-4043-8ba2-fee8d84a0860" (UID: "2aefdb23-64c9-4043-8ba2-fee8d84a0860"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:33.007147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.007088 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-kubeconfig" pod "2aefdb23-64c9-4043-8ba2-fee8d84a0860" (UID: "2aefdb23-64c9-4043-8ba2-fee8d84a0860"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:33.009730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.009687 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aefdb23-64c9-4043-8ba2-fee8d84a0860-local-certs" pod "2aefdb23-64c9-4043-8ba2-fee8d84a0860" (UID: "2aefdb23-64c9-4043-8ba2-fee8d84a0860"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:44:33.010423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.009809 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-token" pod "2aefdb23-64c9-4043-8ba2-fee8d84a0860" (UID: "2aefdb23-64c9-4043-8ba2-fee8d84a0860"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:33.010423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.009927 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-dns-cert" pod "2aefdb23-64c9-4043-8ba2-fee8d84a0860" (UID: "2aefdb23-64c9-4043-8ba2-fee8d84a0860"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:33.014471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.014426 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-cacerts" pod "2aefdb23-64c9-4043-8ba2-fee8d84a0860" (UID: "2aefdb23-64c9-4043-8ba2-fee8d84a0860"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:33.015823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.015786 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-kube-api-access-jqpm9" pod "2aefdb23-64c9-4043-8ba2-fee8d84a0860" (UID: "2aefdb23-64c9-4043-8ba2-fee8d84a0860"). InnerVolumeSpecName "kube-api-access-jqpm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:33.107647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.107451 4294 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-kubeconfig\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:33.107647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.107485 4294 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-cacerts\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:33.107647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.107499 4294 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2aefdb23-64c9-4043-8ba2-fee8d84a0860-local-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:33.107647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.107515 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqpm9\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-kube-api-access-jqpm9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:33.107647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.107532 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:33.107647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.107546 4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-ca-configmap\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:33.107647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.107564 4294 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2aefdb23-64c9-4043-8ba2-fee8d84a0860-istio-csr-dns-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:33.169678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.169648 4294 generic.go:356] "Generic (PLEG): container finished" podID="2aefdb23-64c9-4043-8ba2-fee8d84a0860" containerID="c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645" exitCode=0 Oct 31 21:44:33.169840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.169698 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" Oct 31 21:44:33.169840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.169732 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" event={"ID":"2aefdb23-64c9-4043-8ba2-fee8d84a0860","Type":"ContainerDied","Data":"c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645"} Oct 31 21:44:33.169840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.169786 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj" event={"ID":"2aefdb23-64c9-4043-8ba2-fee8d84a0860","Type":"ContainerDied","Data":"df3b57d83ee28488ad67f5ee27fe54449dfb22e3eb8c7db1077a4e3edf59aff0"} Oct 31 21:44:33.169840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.169813 4294 scope.go:122] "RemoveContainer" containerID="c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645" Oct 31 21:44:33.211030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.211007 4294 scope.go:122] "RemoveContainer" containerID="c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645" Oct 31 21:44:33.211370 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:33.211343 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645\": container with ID starting with c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645 not found: ID does not exist" containerID="c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645" Oct 31 21:44:33.211455 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.211385 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645"} err="failed to get container status \"c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645\": rpc error: code = NotFound desc = could not find container \"c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645\": container with ID starting with c24ad1505d06a30e4e3870a9cc15b034a8e0a1ad39ae76bddd86e23b51ce0645 not found: ID does not exist" Oct 31 21:44:33.247702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.247668 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj"] Oct 31 21:44:33.254083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.254061 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-kb2gj"] Oct 31 21:44:33.588773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.588723 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:44:33.589411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:33.589381 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 21:44:34.004585 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:34.004546 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aefdb23-64c9-4043-8ba2-fee8d84a0860" path="/var/lib/kubelet/pods/2aefdb23-64c9-4043-8ba2-fee8d84a0860/volumes" Oct 31 21:44:34.596442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:34.596404 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body= Oct 31 21:44:34.596850 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:34.596464 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" Oct 31 21:44:34.598570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:34.598543 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:44:35.591268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:35.591236 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:44:35.592192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:35.592173 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:44:37.397902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:37.397867 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw"] Oct 31 21:44:37.435454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:37.435420 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj"] Oct 31 21:44:38.444645 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.444613 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7"] Oct 31 21:44:38.445128 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445114 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" Oct 31 21:44:38.445171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445129 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ef5b225-efbc-4f97-b9b7-300c9c97e9d0" containerName="istio-proxy" Oct 31 21:44:38.445171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445149 4294 cpu_manager.go:414] "containerMap: removing container" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" Oct 31 21:44:38.445171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445154 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" Oct 31 21:44:38.445171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445167 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2aefdb23-64c9-4043-8ba2-fee8d84a0860" containerName="discovery" Oct 31 21:44:38.445295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445172 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aefdb23-64c9-4043-8ba2-fee8d84a0860" containerName="discovery" Oct 31 21:44:38.445295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445262 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ba5bd9-d03d-4399-a8b7-173694f0a66f" containerName="istio-proxy" Oct 31 21:44:38.445295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.445273 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aefdb23-64c9-4043-8ba2-fee8d84a0860" containerName="discovery" Oct 31 21:44:38.448336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.448293 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.478002 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.477963 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-448b01-serving-cert\"" Oct 31 21:44:38.482472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.482443 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7"] Oct 31 21:44:38.587116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.587076 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-maas-api-tls\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.587330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.587152 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-metrics-certs\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.587330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.587236 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c8eec-1cb8-447a-9c8a-e4671aabe194-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.587330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.587296 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mjc\" (UniqueName: \"kubernetes.io/projected/d00c8eec-1cb8-447a-9c8a-e4671aabe194-kube-api-access-v4mjc\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.690532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.689905 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-maas-api-tls\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.690738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.690636 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-metrics-certs\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.690738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.690714 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c8eec-1cb8-447a-9c8a-e4671aabe194-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.690892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.690837 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mjc\" (UniqueName: \"kubernetes.io/projected/d00c8eec-1cb8-447a-9c8a-e4671aabe194-kube-api-access-v4mjc\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.692695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.692636 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c8eec-1cb8-447a-9c8a-e4671aabe194-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.694456 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.694430 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-metrics-certs\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.694956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.694903 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-maas-api-tls\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.716557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.716522 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mjc\" (UniqueName: \"kubernetes.io/projected/d00c8eec-1cb8-447a-9c8a-e4671aabe194-kube-api-access-v4mjc\") pod \"maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:38.760104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:38.760027 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:39.064270 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:44:39.064237 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00c8eec_1cb8_447a_9c8a_e4671aabe194.slice/crio-2c3c7322cacff025f7d574e444e6f1c8bacaef14f3ccfbad714574fe60d3363d WatchSource:0}: Error finding container 2c3c7322cacff025f7d574e444e6f1c8bacaef14f3ccfbad714574fe60d3363d: Status 404 returned error can't find the container with id 2c3c7322cacff025f7d574e444e6f1c8bacaef14f3ccfbad714574fe60d3363d Oct 31 21:44:39.064415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:39.064389 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7"] Oct 31 21:44:39.065803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:39.065783 4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:44:39.092851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:39.092820 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:39.105388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:39.105361 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:39.238514 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:39.238470 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" event={"ID":"d00c8eec-1cb8-447a-9c8a-e4671aabe194","Type":"ContainerStarted","Data":"2c3c7322cacff025f7d574e444e6f1c8bacaef14f3ccfbad714574fe60d3363d"} Oct 31 21:44:39.266373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:39.266289 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:44:40.246505 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:40.246468 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" event={"ID":"d00c8eec-1cb8-447a-9c8a-e4671aabe194","Type":"ContainerStarted","Data":"e692c643163ec91d7784beef46677b922facf436386ad1613d5445b2a020b6eb"} Oct 31 21:44:40.291064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:40.291014 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" podStartSLOduration=1.921786418 podStartE2EDuration="2.290999272s" totalImagesPullingTime="369.212854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:40.288764458 +0000 UTC m=+1720.413871690" watchObservedRunningTime="2026-10-31 21:44:40.290999272 +0000 UTC m=+1720.416106545" Oct 31 21:44:41.252514 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.252482 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:41.392260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.392228 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n"] Oct 31 21:44:41.395139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.395121 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.411336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.411212 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:44:41.422970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.422933 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:44:41.427493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.427466 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n"] Oct 31 21:44:41.526612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.526527 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.526612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.526591 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv9l7\" (UniqueName: \"kubernetes.io/projected/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-kube-api-access-lv9l7\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.526837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.526764 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.627908 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.627872 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv9l7\" (UniqueName: \"kubernetes.io/projected/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-kube-api-access-lv9l7\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.628094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.627958 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.628094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.628057 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.628668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.628646 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.630434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.630411 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.651583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.651551 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv9l7\" (UniqueName: \"kubernetes.io/projected/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-kube-api-access-lv9l7\") pod \"maas-api-revoke-sub-56d321a4-x696n\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:41.709041 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:41.709004 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:42.014601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.014572 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n"] Oct 31 21:44:42.016395 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:44:42.016365 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f4d39d_d8fe_4ae6_9541_e3c9981213d6.slice/crio-2b17428a1f9c43db4ae95afeb09f900d65a94e28b88eb85f839aeab95a31e6ce WatchSource:0}: Error finding container 2b17428a1f9c43db4ae95afeb09f900d65a94e28b88eb85f839aeab95a31e6ce: Status 404 returned error can't find the container with id 2b17428a1f9c43db4ae95afeb09f900d65a94e28b88eb85f839aeab95a31e6ce Oct 31 21:44:42.261771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.261726 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" event={"ID":"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6","Type":"ContainerStarted","Data":"2b17428a1f9c43db4ae95afeb09f900d65a94e28b88eb85f839aeab95a31e6ce"} Oct 31 21:44:42.398950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.398897 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" podUID="a82c25f0-7237-4aa9-ad16-16b19c89a321" containerName="payload-processing" containerID="cri-o://f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6" gracePeriod=25 Oct 31 21:44:42.435873 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.435816 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" podUID="57af7807-56f5-4bc3-9ce5-6e75b541ba05" containerName="payload-pre-processing" containerID="cri-o://914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead" gracePeriod=25 Oct 31 21:44:42.752322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.752274 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:44:42.794407 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.794331 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:44:42.849665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.849629 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a82c25f0-7237-4aa9-ad16-16b19c89a321-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82c25f0-7237-4aa9-ad16-16b19c89a321-plugins-config-volume\") pod \"a82c25f0-7237-4aa9-ad16-16b19c89a321\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " Oct 31 21:44:42.849844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.849788 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a82c25f0-7237-4aa9-ad16-16b19c89a321-kube-api-access-m4srn\" (UniqueName: \"kubernetes.io/projected/a82c25f0-7237-4aa9-ad16-16b19c89a321-kube-api-access-m4srn\") pod \"a82c25f0-7237-4aa9-ad16-16b19c89a321\" (UID: \"a82c25f0-7237-4aa9-ad16-16b19c89a321\") " Oct 31 21:44:42.850203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.850173 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a82c25f0-7237-4aa9-ad16-16b19c89a321-plugins-config-volume" pod "a82c25f0-7237-4aa9-ad16-16b19c89a321" (UID: "a82c25f0-7237-4aa9-ad16-16b19c89a321"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:42.850439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.850417 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a82c25f0-7237-4aa9-ad16-16b19c89a321-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.852824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.852795 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82c25f0-7237-4aa9-ad16-16b19c89a321-kube-api-access-m4srn" pod "a82c25f0-7237-4aa9-ad16-16b19c89a321" (UID: "a82c25f0-7237-4aa9-ad16-16b19c89a321"). InnerVolumeSpecName "kube-api-access-m4srn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:42.951534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.951494 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57af7807-56f5-4bc3-9ce5-6e75b541ba05-kube-api-access-hs2tv\" (UniqueName: \"kubernetes.io/projected/57af7807-56f5-4bc3-9ce5-6e75b541ba05-kube-api-access-hs2tv\") pod \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " Oct 31 21:44:42.951769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.951744 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57af7807-56f5-4bc3-9ce5-6e75b541ba05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af7807-56f5-4bc3-9ce5-6e75b541ba05-plugins-config-volume\") pod \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\" (UID: \"57af7807-56f5-4bc3-9ce5-6e75b541ba05\") " Oct 31 21:44:42.952165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.952139 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57af7807-56f5-4bc3-9ce5-6e75b541ba05-plugins-config-volume" pod "57af7807-56f5-4bc3-9ce5-6e75b541ba05" (UID: "57af7807-56f5-4bc3-9ce5-6e75b541ba05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:42.952285 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.952275 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57af7807-56f5-4bc3-9ce5-6e75b541ba05-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.952379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.952290 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4srn\" (UniqueName: \"kubernetes.io/projected/a82c25f0-7237-4aa9-ad16-16b19c89a321-kube-api-access-m4srn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:42.953605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:42.953581 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57af7807-56f5-4bc3-9ce5-6e75b541ba05-kube-api-access-hs2tv" pod "57af7807-56f5-4bc3-9ce5-6e75b541ba05" (UID: "57af7807-56f5-4bc3-9ce5-6e75b541ba05"). InnerVolumeSpecName "kube-api-access-hs2tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:43.053278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.053189 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2tv\" (UniqueName: \"kubernetes.io/projected/57af7807-56f5-4bc3-9ce5-6e75b541ba05-kube-api-access-hs2tv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:43.269801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.269761 4294 generic.go:356] "Generic (PLEG): container finished" podID="a82c25f0-7237-4aa9-ad16-16b19c89a321" containerID="f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6" exitCode=0 Oct 31 21:44:43.270252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.269859 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" Oct 31 21:44:43.270252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.269869 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" event={"ID":"a82c25f0-7237-4aa9-ad16-16b19c89a321","Type":"ContainerDied","Data":"f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6"} Oct 31 21:44:43.270252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.269922 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw" event={"ID":"a82c25f0-7237-4aa9-ad16-16b19c89a321","Type":"ContainerDied","Data":"6947e145a104bab5b71b41379e312f2ddbc0f1b7ced20bea1fd72ced1ddacb13"} Oct 31 21:44:43.270252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.269952 4294 scope.go:122] "RemoveContainer" containerID="f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6" Oct 31 21:44:43.271331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.271286 4294 generic.go:356] "Generic (PLEG): container finished" podID="57af7807-56f5-4bc3-9ce5-6e75b541ba05" containerID="914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead" exitCode=0 Oct 31 21:44:43.271469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.271447 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" Oct 31 21:44:43.271769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.271747 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" event={"ID":"57af7807-56f5-4bc3-9ce5-6e75b541ba05","Type":"ContainerDied","Data":"914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead"} Oct 31 21:44:43.271851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.271800 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj" event={"ID":"57af7807-56f5-4bc3-9ce5-6e75b541ba05","Type":"ContainerDied","Data":"a829fe912b43ace31402513166cbda9c0eced4e361640c555db9d8a7a7f91dad"} Oct 31 21:44:43.273627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.273604 4294 generic.go:356] "Generic (PLEG): container finished" podID="c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" containerID="4f84f9e432feffca44960e81b23b16f23cfd4b09d2da7a32c36a25f8efc792d5" exitCode=0 Oct 31 21:44:43.273780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.273765 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" event={"ID":"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6","Type":"ContainerDied","Data":"4f84f9e432feffca44960e81b23b16f23cfd4b09d2da7a32c36a25f8efc792d5"} Oct 31 21:44:43.282647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.282632 4294 scope.go:122] "RemoveContainer" containerID="f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6" Oct 31 21:44:43.282914 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:43.282896 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6\": container with ID starting with f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6 not found: ID does not exist" containerID="f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6" Oct 31 21:44:43.282965 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.282922 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6"} err="failed to get container status \"f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6\": rpc error: code = NotFound desc = could not find container \"f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6\": container with ID starting with f3a6c47995183bdab26c2a221ca7a534bfdae643d48c19466c57b55c1f6c20a6 not found: ID does not exist" Oct 31 21:44:43.282965 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.282937 4294 scope.go:122] "RemoveContainer" containerID="914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead" Oct 31 21:44:43.293832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.293814 4294 scope.go:122] "RemoveContainer" containerID="914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead" Oct 31 21:44:43.294150 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:44:43.294129 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead\": container with ID starting with 914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead not found: ID does not exist" containerID="914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead" Oct 31 21:44:43.294202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.294157 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead"} err="failed to get container status \"914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead\": rpc error: code = NotFound desc = could not find container \"914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead\": container with ID starting with 914f6848e30d167ff7bf6778de49a275fa8ab3d63ada1c73902f7d6dcb254ead not found: ID does not exist" Oct 31 21:44:43.328432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.328342 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw"] Oct 31 21:44:43.350503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.350470 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-09f39e-ccff47f95-b9rsw"] Oct 31 21:44:43.489963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.489934 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj"] Oct 31 21:44:43.497090 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.497060 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-09f39e-7c6d78d67f-bmcdj"] Oct 31 21:44:43.986021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.985991 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57af7807-56f5-4bc3-9ce5-6e75b541ba05" path="/var/lib/kubelet/pods/57af7807-56f5-4bc3-9ce5-6e75b541ba05/volumes" Oct 31 21:44:43.986612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:43.986594 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a82c25f0-7237-4aa9-ad16-16b19c89a321" path="/var/lib/kubelet/pods/a82c25f0-7237-4aa9-ad16-16b19c89a321/volumes" Oct 31 21:44:44.416636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.416609 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:44.573643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.573559 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-openshift-ca\") pod \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " Oct 31 21:44:44.573797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.573777 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-xks-ca\") pod \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " Oct 31 21:44:44.573876 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.573816 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-kube-api-access-lv9l7\" (UniqueName: \"kubernetes.io/projected/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-kube-api-access-lv9l7\") pod \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\" (UID: \"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6\") " Oct 31 21:44:44.573923 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.573892 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-openshift-ca" pod "c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" (UID: "c8f4d39d-d8fe-4ae6-9541-e3c9981213d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:44.574283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.574264 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:44.575710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.575684 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-xks-ca" pod "c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" (UID: "c8f4d39d-d8fe-4ae6-9541-e3c9981213d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:44.575809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.575750 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-kube-api-access-lv9l7" pod "c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" (UID: "c8f4d39d-d8fe-4ae6-9541-e3c9981213d6"). InnerVolumeSpecName "kube-api-access-lv9l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:44.675146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.675113 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:44.675146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:44.675146 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv9l7\" (UniqueName: \"kubernetes.io/projected/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6-kube-api-access-lv9l7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:45.288987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:45.288952 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" Oct 31 21:44:45.288987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:45.288980 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n" event={"ID":"c8f4d39d-d8fe-4ae6-9541-e3c9981213d6","Type":"ContainerDied","Data":"2b17428a1f9c43db4ae95afeb09f900d65a94e28b88eb85f839aeab95a31e6ce"} Oct 31 21:44:45.289168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:45.289026 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b17428a1f9c43db4ae95afeb09f900d65a94e28b88eb85f839aeab95a31e6ce" Oct 31 21:44:47.268282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:47.268254 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:44:48.038482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:48.038446 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm"] Oct 31 21:44:48.040951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:48.040929 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-1d8f7b-4b6rm"] Oct 31 21:44:49.984510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:49.984478 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafd9e79-0153-473e-b0ad-0bf8771f370b" path="/var/lib/kubelet/pods/bafd9e79-0153-473e-b0ad-0bf8771f370b/volumes" Oct 31 21:44:55.584258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584220 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4"] Oct 31 21:44:55.584744 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584730 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a82c25f0-7237-4aa9-ad16-16b19c89a321" containerName="payload-processing" Oct 31 21:44:55.584784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584745 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a82c25f0-7237-4aa9-ad16-16b19c89a321" containerName="payload-processing" Oct 31 21:44:55.584784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584760 4294 cpu_manager.go:414] "containerMap: removing container" podUID="57af7807-56f5-4bc3-9ce5-6e75b541ba05" containerName="payload-pre-processing" Oct 31 21:44:55.584784 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584766 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57af7807-56f5-4bc3-9ce5-6e75b541ba05" containerName="payload-pre-processing" Oct 31 21:44:55.584877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584804 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" containerName="revoke-keys" Oct 31 21:44:55.584877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584809 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" containerName="revoke-keys" Oct 31 21:44:55.584934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584895 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a82c25f0-7237-4aa9-ad16-16b19c89a321" containerName="payload-processing" Oct 31 21:44:55.584934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584905 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="57af7807-56f5-4bc3-9ce5-6e75b541ba05" containerName="payload-pre-processing" Oct 31 21:44:55.584934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.584915 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" containerName="revoke-keys" Oct 31 21:44:55.588233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.588218 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.606360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.606334 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4"] Oct 31 21:44:55.614837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.614812 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:44:55.614837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.614836 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:44:55.622343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.622287 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmphf\" (UniqueName: \"kubernetes.io/projected/88a903c3-b760-418b-a5b2-8b3df60ff42c-kube-api-access-gmphf\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.622477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.622458 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.622652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.622632 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.723933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.723897 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.724092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.723987 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmphf\" (UniqueName: \"kubernetes.io/projected/88a903c3-b760-418b-a5b2-8b3df60ff42c-kube-api-access-gmphf\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.724167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.724089 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.724717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.724690 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.726266 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.726246 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.747310 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.747269 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmphf\" (UniqueName: \"kubernetes.io/projected/88a903c3-b760-418b-a5b2-8b3df60ff42c-kube-api-access-gmphf\") pod \"maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:55.900844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:55.900764 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:56.201038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:56.201009 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4"] Oct 31 21:44:56.202551 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:44:56.202516 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a903c3_b760_418b_a5b2_8b3df60ff42c.slice/crio-f16a64780c16f8496a7e51a78bca5fbc56d41b983ce1bc6f6c62901624b0611a WatchSource:0}: Error finding container f16a64780c16f8496a7e51a78bca5fbc56d41b983ce1bc6f6c62901624b0611a: Status 404 returned error can't find the container with id f16a64780c16f8496a7e51a78bca5fbc56d41b983ce1bc6f6c62901624b0611a Oct 31 21:44:56.360288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:56.360253 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" event={"ID":"88a903c3-b760-418b-a5b2-8b3df60ff42c","Type":"ContainerStarted","Data":"f16a64780c16f8496a7e51a78bca5fbc56d41b983ce1bc6f6c62901624b0611a"} Oct 31 21:44:57.368719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:57.368642 4294 generic.go:356] "Generic (PLEG): container finished" podID="88a903c3-b760-418b-a5b2-8b3df60ff42c" containerID="1bd3546e070d6caa881ebd8a0335ce9944f544e36a32b3a07a512f367721eb88" exitCode=0 Oct 31 21:44:57.368719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:57.368704 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" event={"ID":"88a903c3-b760-418b-a5b2-8b3df60ff42c","Type":"ContainerDied","Data":"1bd3546e070d6caa881ebd8a0335ce9944f544e36a32b3a07a512f367721eb88"} Oct 31 21:44:58.348608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.348576 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww"] Oct 31 21:44:58.352554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.352533 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.373464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.373438 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww"] Oct 31 21:44:58.460347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.460103 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.460347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.460158 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.460347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.460183 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkrp8\" (UniqueName: \"kubernetes.io/projected/82101bf8-5865-40b9-b9a3-fa58ded9fa45-kube-api-access-wkrp8\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.568326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.565166 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.568326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.565221 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.568326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.565246 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkrp8\" (UniqueName: \"kubernetes.io/projected/82101bf8-5865-40b9-b9a3-fa58ded9fa45-kube-api-access-wkrp8\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.568326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.566497 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.572593 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.571256 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.589527 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.589498 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkrp8\" (UniqueName: \"kubernetes.io/projected/82101bf8-5865-40b9-b9a3-fa58ded9fa45-kube-api-access-wkrp8\") pod \"maas-api-revoke-sub-ba3a8207-vtbww\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.625184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.625162 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:58.665506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.665479 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:44:58.667137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.666981 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a903c3-b760-418b-a5b2-8b3df60ff42c-kube-api-access-gmphf\" (UniqueName: \"kubernetes.io/projected/88a903c3-b760-418b-a5b2-8b3df60ff42c-kube-api-access-gmphf\") pod \"88a903c3-b760-418b-a5b2-8b3df60ff42c\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " Oct 31 21:44:58.667137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.667054 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-xks-ca\") pod \"88a903c3-b760-418b-a5b2-8b3df60ff42c\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " Oct 31 21:44:58.667137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.667100 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-openshift-ca\") pod \"88a903c3-b760-418b-a5b2-8b3df60ff42c\" (UID: \"88a903c3-b760-418b-a5b2-8b3df60ff42c\") " Oct 31 21:44:58.668018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.667993 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-openshift-ca" pod "88a903c3-b760-418b-a5b2-8b3df60ff42c" (UID: "88a903c3-b760-418b-a5b2-8b3df60ff42c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:44:58.675794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.675759 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a903c3-b760-418b-a5b2-8b3df60ff42c-kube-api-access-gmphf" pod "88a903c3-b760-418b-a5b2-8b3df60ff42c" (UID: "88a903c3-b760-418b-a5b2-8b3df60ff42c"). InnerVolumeSpecName "kube-api-access-gmphf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:44:58.675794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.675774 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-xks-ca" pod "88a903c3-b760-418b-a5b2-8b3df60ff42c" (UID: "88a903c3-b760-418b-a5b2-8b3df60ff42c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:44:58.768614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.768580 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmphf\" (UniqueName: \"kubernetes.io/projected/88a903c3-b760-418b-a5b2-8b3df60ff42c-kube-api-access-gmphf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:58.768614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.768611 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:58.768614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.768622 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88a903c3-b760-418b-a5b2-8b3df60ff42c-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:44:58.975592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:58.975562 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww"] Oct 31 21:44:58.977409 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:44:58.977379 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82101bf8_5865_40b9_b9a3_fa58ded9fa45.slice/crio-fca77af4f35260d49bffd4688f1e5ff2ef9a684e553c0923563eb0a3c8b3f85c WatchSource:0}: Error finding container fca77af4f35260d49bffd4688f1e5ff2ef9a684e553c0923563eb0a3c8b3f85c: Status 404 returned error can't find the container with id fca77af4f35260d49bffd4688f1e5ff2ef9a684e553c0923563eb0a3c8b3f85c Oct 31 21:44:59.385433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:59.385395 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" event={"ID":"82101bf8-5865-40b9-b9a3-fa58ded9fa45","Type":"ContainerStarted","Data":"1c1f0dec6ed4342254fd2736848b850ee296ed95a8b0890a7c83ceeadb01563c"} Oct 31 21:44:59.385875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:59.385495 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" event={"ID":"82101bf8-5865-40b9-b9a3-fa58ded9fa45","Type":"ContainerStarted","Data":"fca77af4f35260d49bffd4688f1e5ff2ef9a684e553c0923563eb0a3c8b3f85c"} Oct 31 21:44:59.386927 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:59.386902 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" event={"ID":"88a903c3-b760-418b-a5b2-8b3df60ff42c","Type":"ContainerDied","Data":"f16a64780c16f8496a7e51a78bca5fbc56d41b983ce1bc6f6c62901624b0611a"} Oct 31 21:44:59.387034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:59.386945 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f16a64780c16f8496a7e51a78bca5fbc56d41b983ce1bc6f6c62901624b0611a" Oct 31 21:44:59.387034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:59.386909 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4" Oct 31 21:44:59.432941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:44:59.432890 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" podStartSLOduration=1.4328758750000001 podStartE2EDuration="1.432875875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:44:59.431775364 +0000 UTC m=+1739.556882593" watchObservedRunningTime="2026-10-31 21:44:59.432875875 +0000 UTC m=+1739.557983123" Oct 31 21:45:00.147042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.146995 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh"] Oct 31 21:45:00.148229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.148198 4294 cpu_manager.go:414] "containerMap: removing container" podUID="88a903c3-b760-418b-a5b2-8b3df60ff42c" containerName="revoke-keys" Oct 31 21:45:00.148400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.148232 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a903c3-b760-418b-a5b2-8b3df60ff42c" containerName="revoke-keys" Oct 31 21:45:00.148493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.148438 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a903c3-b760-418b-a5b2-8b3df60ff42c" containerName="revoke-keys" Oct 31 21:45:00.152282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.152260 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5"] Oct 31 21:45:00.152424 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.152407 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.156936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.156910 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx"] Oct 31 21:45:00.157052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.157025 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.162839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.162413 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.177079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.177027 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh"] Oct 31 21:45:00.190753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.190727 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv"] Oct 31 21:45:00.195026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.194995 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7cebf9-0de6-4217-9913-4d1aea45ff94-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891385-j4bgh\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.195158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.195134 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fqc\" (UniqueName: \"kubernetes.io/projected/5d7cebf9-0de6-4217-9913-4d1aea45ff94-kube-api-access-v4fqc\") pod \"maas-api-key-cleanup-29891385-j4bgh\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.220244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.220211 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j"] Oct 31 21:45:00.220542 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.220517 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.226076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.226050 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx"] Oct 31 21:45:00.226225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.226098 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5"] Oct 31 21:45:00.226225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.226126 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv"] Oct 31 21:45:00.226225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.226152 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz"] Oct 31 21:45:00.227521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.227363 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.247186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.246896 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:00.250296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.250272 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j"] Oct 31 21:45:00.254396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.254345 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz"] Oct 31 21:45:00.296865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.296814 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvplm\" (UniqueName: \"kubernetes.io/projected/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-kube-api-access-zvplm\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.297068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.296949 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a577fdb8-0da4-41ae-806f-f3e463e3877e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.297068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.296984 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7cebf9-0de6-4217-9913-4d1aea45ff94-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891385-j4bgh\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.297068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297033 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrlb\" (UniqueName: \"kubernetes.io/projected/a577fdb8-0da4-41ae-806f-f3e463e3877e-kube-api-access-pmrlb\") pod \"maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.297260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297163 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a738352-7a54-4d5a-8594-1b960a43705d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:00.297260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297198 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.297260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297219 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fqc\" (UniqueName: \"kubernetes.io/projected/5d7cebf9-0de6-4217-9913-4d1aea45ff94-kube-api-access-v4fqc\") pod \"maas-api-key-cleanup-29891385-j4bgh\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.297480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297256 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlskw\" (UniqueName: \"kubernetes.io/projected/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-kube-api-access-vlskw\") pod \"maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.297688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297417 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn9h4\" (UniqueName: \"kubernetes.io/projected/7780c66d-2d0f-4a3d-9131-a41605700f80-kube-api-access-nn9h4\") pod \"maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.297762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297728 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780c66d-2d0f-4a3d-9131-a41605700f80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.298045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297760 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.298045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297812 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7cebf9-0de6-4217-9913-4d1aea45ff94-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891385-j4bgh\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.298045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.297889 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn94t\" (UniqueName: \"kubernetes.io/projected/9a738352-7a54-4d5a-8594-1b960a43705d-kube-api-access-jn94t\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:00.330160 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.327589 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fqc\" (UniqueName: \"kubernetes.io/projected/5d7cebf9-0de6-4217-9913-4d1aea45ff94-kube-api-access-v4fqc\") pod \"maas-api-key-cleanup-29891385-j4bgh\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.403339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403187 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn94t\" (UniqueName: \"kubernetes.io/projected/9a738352-7a54-4d5a-8594-1b960a43705d-kube-api-access-jn94t\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:00.403339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403259 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvplm\" (UniqueName: \"kubernetes.io/projected/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-kube-api-access-zvplm\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403344 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a577fdb8-0da4-41ae-806f-f3e463e3877e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403428 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrlb\" (UniqueName: \"kubernetes.io/projected/a577fdb8-0da4-41ae-806f-f3e463e3877e-kube-api-access-pmrlb\") pod \"maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403503 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a738352-7a54-4d5a-8594-1b960a43705d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403541 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403594 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlskw\" (UniqueName: \"kubernetes.io/projected/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-kube-api-access-vlskw\") pod \"maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403643 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn9h4\" (UniqueName: \"kubernetes.io/projected/7780c66d-2d0f-4a3d-9131-a41605700f80-kube-api-access-nn9h4\") pod \"maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403682 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780c66d-2d0f-4a3d-9131-a41605700f80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.403911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.403707 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.414209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.404969 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a738352-7a54-4d5a-8594-1b960a43705d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:00.414209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.406025 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a577fdb8-0da4-41ae-806f-f3e463e3877e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.414209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.407356 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.414209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.409181 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780c66d-2d0f-4a3d-9131-a41605700f80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.414209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.409583 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.414209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.412704 4294 generic.go:356] "Generic (PLEG): container finished" podID="82101bf8-5865-40b9-b9a3-fa58ded9fa45" containerID="1c1f0dec6ed4342254fd2736848b850ee296ed95a8b0890a7c83ceeadb01563c" exitCode=0 Oct 31 21:45:00.414209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.412749 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" event={"ID":"82101bf8-5865-40b9-b9a3-fa58ded9fa45","Type":"ContainerDied","Data":"1c1f0dec6ed4342254fd2736848b850ee296ed95a8b0890a7c83ceeadb01563c"} Oct 31 21:45:00.450634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.439811 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn9h4\" (UniqueName: \"kubernetes.io/projected/7780c66d-2d0f-4a3d-9131-a41605700f80-kube-api-access-nn9h4\") pod \"maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.450634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.441407 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlskw\" (UniqueName: \"kubernetes.io/projected/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-kube-api-access-vlskw\") pod \"maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.450634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.450260 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrlb\" (UniqueName: \"kubernetes.io/projected/a577fdb8-0da4-41ae-806f-f3e463e3877e-kube-api-access-pmrlb\") pod \"maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.470063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.454786 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvplm\" (UniqueName: \"kubernetes.io/projected/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-kube-api-access-zvplm\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.470063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.455404 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn94t\" (UniqueName: \"kubernetes.io/projected/9a738352-7a54-4d5a-8594-1b960a43705d-kube-api-access-jn94t\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:00.470532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.470288 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:00.525091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.519567 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:00.529906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.528934 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:00.548368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.548187 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:00.565194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.564040 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:00.626736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:00.576718 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:01.160349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.159778 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh"] Oct 31 21:45:01.173353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.171570 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx"] Oct 31 21:45:01.253508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.253448 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5"] Oct 31 21:45:01.263521 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:01.263491 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d494a4_63cc_4d4d_9fc0_6c00bec5ece2.slice/crio-0c17fa28f8b1f6663c3d77016a4d363b7c528c59b7d47f9b306b75cafd648686 WatchSource:0}: Error finding container 0c17fa28f8b1f6663c3d77016a4d363b7c528c59b7d47f9b306b75cafd648686: Status 404 returned error can't find the container with id 0c17fa28f8b1f6663c3d77016a4d363b7c528c59b7d47f9b306b75cafd648686 Oct 31 21:45:01.347072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.346754 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j"] Oct 31 21:45:01.366402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.364729 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv"] Oct 31 21:45:01.374491 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:01.373708 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:45:01.378469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.378291 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz"] Oct 31 21:45:01.384122 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:01.384095 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a738352_7a54_4d5a_8594_1b960a43705d.slice/crio-73b1e5edaad0a3515046582d6845a7cfa809239d5e12e4b21b6299fb6f56b6e7 WatchSource:0}: Error finding container 73b1e5edaad0a3515046582d6845a7cfa809239d5e12e4b21b6299fb6f56b6e7: Status 404 returned error can't find the container with id 73b1e5edaad0a3515046582d6845a7cfa809239d5e12e4b21b6299fb6f56b6e7 Oct 31 21:45:01.420063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.420036 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" event={"ID":"a577fdb8-0da4-41ae-806f-f3e463e3877e","Type":"ContainerStarted","Data":"4f7553b9d7e94fc54b51313477d6bc24d8412df457f30974559bf104a23f3779"} Oct 31 21:45:01.421378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.421358 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" event={"ID":"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2","Type":"ContainerStarted","Data":"0c17fa28f8b1f6663c3d77016a4d363b7c528c59b7d47f9b306b75cafd648686"} Oct 31 21:45:01.422495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.422471 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" event={"ID":"7780c66d-2d0f-4a3d-9131-a41605700f80","Type":"ContainerStarted","Data":"f749a44d51596da37f0e7cd933482f39e90d8888bc6cbf519d762864baf0f23b"} Oct 31 21:45:01.423658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.423638 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" event={"ID":"5d7cebf9-0de6-4217-9913-4d1aea45ff94","Type":"ContainerStarted","Data":"95affb2253251774d9d39ecba795efdf6625fb91795470be89acb40154cc2c8b"} Oct 31 21:45:01.424824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.424800 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" event={"ID":"9a738352-7a54-4d5a-8594-1b960a43705d","Type":"ContainerStarted","Data":"73b1e5edaad0a3515046582d6845a7cfa809239d5e12e4b21b6299fb6f56b6e7"} Oct 31 21:45:01.425900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.425884 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" event={"ID":"2b02ae2b-fddd-4d57-a320-c7e45808ca4f","Type":"ContainerStarted","Data":"efe55a719386edec2bd22027d1da732f9beb2fe5f7bd866eb024a916b24e6ec1"} Oct 31 21:45:01.632613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.632590 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:45:01.658897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.658866 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82101bf8-5865-40b9-b9a3-fa58ded9fa45-kube-api-access-wkrp8\" (UniqueName: \"kubernetes.io/projected/82101bf8-5865-40b9-b9a3-fa58ded9fa45-kube-api-access-wkrp8\") pod \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " Oct 31 21:45:01.659017 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.658963 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-xks-ca\") pod \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " Oct 31 21:45:01.659136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.659121 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-openshift-ca\") pod \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\" (UID: \"82101bf8-5865-40b9-b9a3-fa58ded9fa45\") " Oct 31 21:45:01.659549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.659517 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-openshift-ca" pod "82101bf8-5865-40b9-b9a3-fa58ded9fa45" (UID: "82101bf8-5865-40b9-b9a3-fa58ded9fa45"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:01.661118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.661077 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-xks-ca" pod "82101bf8-5865-40b9-b9a3-fa58ded9fa45" (UID: "82101bf8-5865-40b9-b9a3-fa58ded9fa45"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:01.661233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.661169 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82101bf8-5865-40b9-b9a3-fa58ded9fa45-kube-api-access-wkrp8" pod "82101bf8-5865-40b9-b9a3-fa58ded9fa45" (UID: "82101bf8-5865-40b9-b9a3-fa58ded9fa45"). InnerVolumeSpecName "kube-api-access-wkrp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:01.760633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.760604 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkrp8\" (UniqueName: \"kubernetes.io/projected/82101bf8-5865-40b9-b9a3-fa58ded9fa45-kube-api-access-wkrp8\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:01.760633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.760637 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:01.760820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:01.760652 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82101bf8-5865-40b9-b9a3-fa58ded9fa45-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:02.434213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.434180 4294 generic.go:356] "Generic (PLEG): container finished" podID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerID="524aeea6d4b8a96af88d6d677dccddc9917cffe39215d72a1d067b1c6e8277c8" exitCode=60 Oct 31 21:45:02.434625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.434261 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" event={"ID":"a577fdb8-0da4-41ae-806f-f3e463e3877e","Type":"ContainerDied","Data":"524aeea6d4b8a96af88d6d677dccddc9917cffe39215d72a1d067b1c6e8277c8"} Oct 31 21:45:02.434625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.434592 4294 scope.go:122] "RemoveContainer" containerID="524aeea6d4b8a96af88d6d677dccddc9917cffe39215d72a1d067b1c6e8277c8" Oct 31 21:45:02.436472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.436453 4294 generic.go:356] "Generic (PLEG): container finished" podID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerID="1f8e15e8b00af2bdaa341d5361cda3afe375f8c0ff6692cc055e7209588e383a" exitCode=60 Oct 31 21:45:02.436582 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.436565 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" event={"ID":"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2","Type":"ContainerDied","Data":"1f8e15e8b00af2bdaa341d5361cda3afe375f8c0ff6692cc055e7209588e383a"} Oct 31 21:45:02.436751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.436735 4294 scope.go:122] "RemoveContainer" containerID="1f8e15e8b00af2bdaa341d5361cda3afe375f8c0ff6692cc055e7209588e383a" Oct 31 21:45:02.438293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.438271 4294 generic.go:356] "Generic (PLEG): container finished" podID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerID="d34946fd9bcfd237e2a73717dc612d78af36fe05624e25d40bf24a6f2cf95d43" exitCode=60 Oct 31 21:45:02.438398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.438329 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" event={"ID":"7780c66d-2d0f-4a3d-9131-a41605700f80","Type":"ContainerDied","Data":"d34946fd9bcfd237e2a73717dc612d78af36fe05624e25d40bf24a6f2cf95d43"} Oct 31 21:45:02.438553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.438539 4294 scope.go:122] "RemoveContainer" containerID="d34946fd9bcfd237e2a73717dc612d78af36fe05624e25d40bf24a6f2cf95d43" Oct 31 21:45:02.439928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.439905 4294 generic.go:356] "Generic (PLEG): container finished" podID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerID="6fe9106ed6d1eb20280e60fcb5f60fd4afaa804d947fc46b8241c94b66cac9b8" exitCode=60 Oct 31 21:45:02.440012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.439986 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" event={"ID":"5d7cebf9-0de6-4217-9913-4d1aea45ff94","Type":"ContainerDied","Data":"6fe9106ed6d1eb20280e60fcb5f60fd4afaa804d947fc46b8241c94b66cac9b8"} Oct 31 21:45:02.440145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.440131 4294 scope.go:122] "RemoveContainer" containerID="6fe9106ed6d1eb20280e60fcb5f60fd4afaa804d947fc46b8241c94b66cac9b8" Oct 31 21:45:02.445723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.445700 4294 generic.go:356] "Generic (PLEG): container finished" podID="9a738352-7a54-4d5a-8594-1b960a43705d" containerID="038312d75ada816cdba692aa384ce216b9da2f39991075fd757a79bdfe6c28f3" exitCode=60 Oct 31 21:45:02.445812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.445788 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" event={"ID":"9a738352-7a54-4d5a-8594-1b960a43705d","Type":"ContainerDied","Data":"038312d75ada816cdba692aa384ce216b9da2f39991075fd757a79bdfe6c28f3"} Oct 31 21:45:02.447829 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.447805 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" event={"ID":"82101bf8-5865-40b9-b9a3-fa58ded9fa45","Type":"ContainerDied","Data":"fca77af4f35260d49bffd4688f1e5ff2ef9a684e553c0923563eb0a3c8b3f85c"} Oct 31 21:45:02.447940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.447836 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww" Oct 31 21:45:02.447940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.447848 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fca77af4f35260d49bffd4688f1e5ff2ef9a684e553c0923563eb0a3c8b3f85c" Oct 31 21:45:02.448726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.448714 4294 scope.go:122] "RemoveContainer" containerID="038312d75ada816cdba692aa384ce216b9da2f39991075fd757a79bdfe6c28f3" Oct 31 21:45:02.449661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.449640 4294 generic.go:356] "Generic (PLEG): container finished" podID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerID="0129a9c85f9df90a735599f7b6f077b8407297099c2f6354badb0b3711cb31bf" exitCode=60 Oct 31 21:45:02.449773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.449667 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" event={"ID":"2b02ae2b-fddd-4d57-a320-c7e45808ca4f","Type":"ContainerDied","Data":"0129a9c85f9df90a735599f7b6f077b8407297099c2f6354badb0b3711cb31bf"} Oct 31 21:45:02.449878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:02.449863 4294 scope.go:122] "RemoveContainer" containerID="0129a9c85f9df90a735599f7b6f077b8407297099c2f6354badb0b3711cb31bf" Oct 31 21:45:03.458798 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.458763 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" event={"ID":"5d7cebf9-0de6-4217-9913-4d1aea45ff94","Type":"ContainerStarted","Data":"de27b9685c9164b6b6826e59dc869f342d96f11b9637fd15e918f2c833da53ed"} Oct 31 21:45:03.460834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.460803 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" event={"ID":"9a738352-7a54-4d5a-8594-1b960a43705d","Type":"ContainerStarted","Data":"36a83edfa018577fc532a8e3a7b97d123fd0ffbd781f6f15606f69653bc3ca66"} Oct 31 21:45:03.462622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.462596 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" event={"ID":"2b02ae2b-fddd-4d57-a320-c7e45808ca4f","Type":"ContainerStarted","Data":"3686e4f7e23ab426dab20dc6d220c218118a639a2d900b968c532e2558029545"} Oct 31 21:45:03.464371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.464342 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" event={"ID":"a577fdb8-0da4-41ae-806f-f3e463e3877e","Type":"ContainerStarted","Data":"0f40d4608ec57a4c970184648484757fb3ee94fe9f9556f6bdd0369ce3b7f107"} Oct 31 21:45:03.466198 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.466176 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" event={"ID":"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2","Type":"ContainerStarted","Data":"d61eb18d74ca2a3f2ff3d5afbac21b422675179eaf3414d39149c1dc39b11c09"} Oct 31 21:45:03.467779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.467759 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" event={"ID":"7780c66d-2d0f-4a3d-9131-a41605700f80","Type":"ContainerStarted","Data":"71ee3fff826b55a17ae40f8df197aa6fecb199cc2ce3537b948db7ce80244f0f"} Oct 31 21:45:03.502592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.502535 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" podStartSLOduration=3.502523357 podStartE2EDuration="3.502523357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.500202382 +0000 UTC m=+1743.625309612" watchObservedRunningTime="2026-10-31 21:45:03.502523357 +0000 UTC m=+1743.627630588" Oct 31 21:45:03.546736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.546690 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" podStartSLOduration=3.546678243 podStartE2EDuration="3.546678243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.544456295 +0000 UTC m=+1743.669563526" watchObservedRunningTime="2026-10-31 21:45:03.546678243 +0000 UTC m=+1743.671785534" Oct 31 21:45:03.629166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.629116 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" podStartSLOduration=3.62910226 podStartE2EDuration="3.62910226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.588484502 +0000 UTC m=+1743.713591734" watchObservedRunningTime="2026-10-31 21:45:03.62910226 +0000 UTC m=+1743.754209490" Oct 31 21:45:03.629981 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.629947 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" podStartSLOduration=3.629939297 podStartE2EDuration="3.629939297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.627975869 +0000 UTC m=+1743.753083099" watchObservedRunningTime="2026-10-31 21:45:03.629939297 +0000 UTC m=+1743.755046529" Oct 31 21:45:03.670747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.670702 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" podStartSLOduration=3.670689172 podStartE2EDuration="3.670689172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.667339136 +0000 UTC m=+1743.792446364" watchObservedRunningTime="2026-10-31 21:45:03.670689172 +0000 UTC m=+1743.795796404" Oct 31 21:45:03.707350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:03.707268 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" podStartSLOduration=3.707252708 podStartE2EDuration="3.707252708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:03.706826578 +0000 UTC m=+1743.831933810" watchObservedRunningTime="2026-10-31 21:45:03.707252708 +0000 UTC m=+1743.832359974" Oct 31 21:45:04.475969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.475937 4294 generic.go:356] "Generic (PLEG): container finished" podID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerID="de27b9685c9164b6b6826e59dc869f342d96f11b9637fd15e918f2c833da53ed" exitCode=60 Oct 31 21:45:04.476419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.476019 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" event={"ID":"5d7cebf9-0de6-4217-9913-4d1aea45ff94","Type":"ContainerDied","Data":"de27b9685c9164b6b6826e59dc869f342d96f11b9637fd15e918f2c833da53ed"} Oct 31 21:45:04.476419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.476083 4294 scope.go:122] "RemoveContainer" containerID="6fe9106ed6d1eb20280e60fcb5f60fd4afaa804d947fc46b8241c94b66cac9b8" Oct 31 21:45:04.476419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.476257 4294 scope.go:122] "RemoveContainer" containerID="de27b9685c9164b6b6826e59dc869f342d96f11b9637fd15e918f2c833da53ed" Oct 31 21:45:04.476606 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:04.476579 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891385-j4bgh_odh-ai-gateway-infra(5d7cebf9-0de6-4217-9913-4d1aea45ff94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" Oct 31 21:45:04.478146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.478126 4294 generic.go:356] "Generic (PLEG): container finished" podID="9a738352-7a54-4d5a-8594-1b960a43705d" containerID="36a83edfa018577fc532a8e3a7b97d123fd0ffbd781f6f15606f69653bc3ca66" exitCode=60 Oct 31 21:45:04.478246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.478183 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" event={"ID":"9a738352-7a54-4d5a-8594-1b960a43705d","Type":"ContainerDied","Data":"36a83edfa018577fc532a8e3a7b97d123fd0ffbd781f6f15606f69653bc3ca66"} Oct 31 21:45:04.478398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.478384 4294 scope.go:122] "RemoveContainer" containerID="36a83edfa018577fc532a8e3a7b97d123fd0ffbd781f6f15606f69653bc3ca66" Oct 31 21:45:04.478664 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:04.478646 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz_odh-ai-gateway-infra(9a738352-7a54-4d5a-8594-1b960a43705d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" Oct 31 21:45:04.480010 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.479924 4294 generic.go:356] "Generic (PLEG): container finished" podID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerID="3686e4f7e23ab426dab20dc6d220c218118a639a2d900b968c532e2558029545" exitCode=60 Oct 31 21:45:04.480100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.480007 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" event={"ID":"2b02ae2b-fddd-4d57-a320-c7e45808ca4f","Type":"ContainerDied","Data":"3686e4f7e23ab426dab20dc6d220c218118a639a2d900b968c532e2558029545"} Oct 31 21:45:04.480194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.480174 4294 scope.go:122] "RemoveContainer" containerID="3686e4f7e23ab426dab20dc6d220c218118a639a2d900b968c532e2558029545" Oct 31 21:45:04.480507 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:04.480477 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx_odh-ai-gateway-infra(2b02ae2b-fddd-4d57-a320-c7e45808ca4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" Oct 31 21:45:04.481818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.481800 4294 generic.go:356] "Generic (PLEG): container finished" podID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerID="0f40d4608ec57a4c970184648484757fb3ee94fe9f9556f6bdd0369ce3b7f107" exitCode=60 Oct 31 21:45:04.481908 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.481825 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" event={"ID":"a577fdb8-0da4-41ae-806f-f3e463e3877e","Type":"ContainerDied","Data":"0f40d4608ec57a4c970184648484757fb3ee94fe9f9556f6bdd0369ce3b7f107"} Oct 31 21:45:04.482102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.482074 4294 scope.go:122] "RemoveContainer" containerID="0f40d4608ec57a4c970184648484757fb3ee94fe9f9556f6bdd0369ce3b7f107" Oct 31 21:45:04.482395 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:04.482370 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv_odh-ai-gateway-infra(a577fdb8-0da4-41ae-806f-f3e463e3877e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" Oct 31 21:45:04.483573 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.483555 4294 generic.go:356] "Generic (PLEG): container finished" podID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerID="d61eb18d74ca2a3f2ff3d5afbac21b422675179eaf3414d39149c1dc39b11c09" exitCode=60 Oct 31 21:45:04.483669 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.483628 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" event={"ID":"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2","Type":"ContainerDied","Data":"d61eb18d74ca2a3f2ff3d5afbac21b422675179eaf3414d39149c1dc39b11c09"} Oct 31 21:45:04.484263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.484246 4294 scope.go:122] "RemoveContainer" containerID="d61eb18d74ca2a3f2ff3d5afbac21b422675179eaf3414d39149c1dc39b11c09" Oct 31 21:45:04.484524 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:04.484503 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5_odh-ai-gateway-infra(d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" Oct 31 21:45:04.485557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.485539 4294 generic.go:356] "Generic (PLEG): container finished" podID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerID="71ee3fff826b55a17ae40f8df197aa6fecb199cc2ce3537b948db7ce80244f0f" exitCode=60 Oct 31 21:45:04.485651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.485575 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" event={"ID":"7780c66d-2d0f-4a3d-9131-a41605700f80","Type":"ContainerDied","Data":"71ee3fff826b55a17ae40f8df197aa6fecb199cc2ce3537b948db7ce80244f0f"} Oct 31 21:45:04.485750 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.485737 4294 scope.go:122] "RemoveContainer" containerID="71ee3fff826b55a17ae40f8df197aa6fecb199cc2ce3537b948db7ce80244f0f" Oct 31 21:45:04.485943 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:04.485925 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j_odh-ai-gateway-infra(7780c66d-2d0f-4a3d-9131-a41605700f80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" Oct 31 21:45:04.489605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.489586 4294 scope.go:122] "RemoveContainer" containerID="038312d75ada816cdba692aa384ce216b9da2f39991075fd757a79bdfe6c28f3" Oct 31 21:45:04.501941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.501922 4294 scope.go:122] "RemoveContainer" containerID="0129a9c85f9df90a735599f7b6f077b8407297099c2f6354badb0b3711cb31bf" Oct 31 21:45:04.514391 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.514372 4294 scope.go:122] "RemoveContainer" containerID="524aeea6d4b8a96af88d6d677dccddc9917cffe39215d72a1d067b1c6e8277c8" Oct 31 21:45:04.527421 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.527403 4294 scope.go:122] "RemoveContainer" containerID="1f8e15e8b00af2bdaa341d5361cda3afe375f8c0ff6692cc055e7209588e383a" Oct 31 21:45:04.539216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:04.539200 4294 scope.go:122] "RemoveContainer" containerID="d34946fd9bcfd237e2a73717dc612d78af36fe05624e25d40bf24a6f2cf95d43" Oct 31 21:45:05.495694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:05.495667 4294 scope.go:122] "RemoveContainer" containerID="0f40d4608ec57a4c970184648484757fb3ee94fe9f9556f6bdd0369ce3b7f107" Oct 31 21:45:05.496155 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:05.495996 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv_odh-ai-gateway-infra(a577fdb8-0da4-41ae-806f-f3e463e3877e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" Oct 31 21:45:05.497150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:05.497132 4294 scope.go:122] "RemoveContainer" containerID="d61eb18d74ca2a3f2ff3d5afbac21b422675179eaf3414d39149c1dc39b11c09" Oct 31 21:45:05.497428 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:05.497406 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5_odh-ai-gateway-infra(d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" Oct 31 21:45:05.498726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:05.498707 4294 scope.go:122] "RemoveContainer" containerID="71ee3fff826b55a17ae40f8df197aa6fecb199cc2ce3537b948db7ce80244f0f" Oct 31 21:45:05.499000 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:05.498976 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j_odh-ai-gateway-infra(7780c66d-2d0f-4a3d-9131-a41605700f80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" Oct 31 21:45:05.500312 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:05.500283 4294 scope.go:122] "RemoveContainer" containerID="de27b9685c9164b6b6826e59dc869f342d96f11b9637fd15e918f2c833da53ed" Oct 31 21:45:05.500524 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:05.500504 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891385-j4bgh_odh-ai-gateway-infra(5d7cebf9-0de6-4217-9913-4d1aea45ff94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" Oct 31 21:45:05.501785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:05.501766 4294 scope.go:122] "RemoveContainer" containerID="36a83edfa018577fc532a8e3a7b97d123fd0ffbd781f6f15606f69653bc3ca66" Oct 31 21:45:05.501996 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:05.501972 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz_odh-ai-gateway-infra(9a738352-7a54-4d5a-8594-1b960a43705d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" Oct 31 21:45:05.503225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:05.503209 4294 scope.go:122] "RemoveContainer" containerID="3686e4f7e23ab426dab20dc6d220c218118a639a2d900b968c532e2558029545" Oct 31 21:45:05.503433 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:05.503415 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx_odh-ai-gateway-infra(2b02ae2b-fddd-4d57-a320-c7e45808ca4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" Oct 31 21:45:10.582416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.582376 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg"] Oct 31 21:45:10.582970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.582955 4294 cpu_manager.go:414] "containerMap: removing container" podUID="82101bf8-5865-40b9-b9a3-fa58ded9fa45" containerName="revoke-keys" Oct 31 21:45:10.583045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.582969 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82101bf8-5865-40b9-b9a3-fa58ded9fa45" containerName="revoke-keys" Oct 31 21:45:10.583104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.583070 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="82101bf8-5865-40b9-b9a3-fa58ded9fa45" containerName="revoke-keys" Oct 31 21:45:10.587925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.587904 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.607425 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.607396 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg"] Oct 31 21:45:10.615026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.614993 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-1f523a-openshift-default-dockercfg-dhm5b\"" Oct 31 21:45:10.676021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.675984 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-istio-token\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676196 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676037 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-credential-socket\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676196 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676135 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-certs\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676193 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-socket\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676253 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134fced6-b695-470e-b923-d59215873872-istio-podinfo\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676389 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-data\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676437 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzhk8\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-kube-api-access-dzhk8\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676565 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676513 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-envoy\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.676621 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.676574 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134fced6-b695-470e-b923-d59215873872-istiod-ca-cert\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779097 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-istio-token\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779171 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-credential-socket\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779214 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-certs\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779264 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-socket\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779289 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134fced6-b695-470e-b923-d59215873872-istio-podinfo\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779396 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-data\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779443 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzhk8\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-kube-api-access-dzhk8\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779475 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-envoy\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.779524 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134fced6-b695-470e-b923-d59215873872-istiod-ca-cert\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.780908 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.780766 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-credential-socket\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.795325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.781003 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-certs\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.795325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.781251 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-socket\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.795325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.787283 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134fced6-b695-470e-b923-d59215873872-istio-podinfo\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.795325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.787555 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-data\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.802996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.802865 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134fced6-b695-470e-b923-d59215873872-istiod-ca-cert\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.813558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.813494 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-envoy\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.815250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.815214 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzhk8\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-kube-api-access-dzhk8\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.833612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.833544 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-istio-token\") pod \"e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:10.905009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:10.904972 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:11.287695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.287666 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg"] Oct 31 21:45:11.289680 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:11.289654 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134fced6_b695_470e_b923_d59215873872.slice/crio-767e752f1cfc9608c4c09be32e1248b73b7f97ba5f66e693132da3cc67476de9 WatchSource:0}: Error finding container 767e752f1cfc9608c4c09be32e1248b73b7f97ba5f66e693132da3cc67476de9: Status 404 returned error can't find the container with id 767e752f1cfc9608c4c09be32e1248b73b7f97ba5f66e693132da3cc67476de9 Oct 31 21:45:11.292290 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.292260 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:45:11.292420 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.292354 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:45:11.292420 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.292387 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:45:11.546804 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.546707 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" event={"ID":"134fced6-b695-470e-b923-d59215873872","Type":"ContainerStarted","Data":"52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb"} Oct 31 21:45:11.546804 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.546770 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" event={"ID":"134fced6-b695-470e-b923-d59215873872","Type":"ContainerStarted","Data":"767e752f1cfc9608c4c09be32e1248b73b7f97ba5f66e693132da3cc67476de9"} Oct 31 21:45:11.550520 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.550492 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99"] Oct 31 21:45:11.554516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.554499 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.574357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.573971 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99"] Oct 31 21:45:11.596937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.596865 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podStartSLOduration=1.596844456 podStartE2EDuration="1.596844456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:11.591190659 +0000 UTC m=+1751.716297890" watchObservedRunningTime="2026-10-31 21:45:11.596844456 +0000 UTC m=+1751.721951688" Oct 31 21:45:11.653476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.653448 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:45:11.692261 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.692231 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.692465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.692448 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.692589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.692561 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhn5\" (UniqueName: \"kubernetes.io/projected/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-kube-api-access-rzhn5\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.794689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.794646 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.794863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.794732 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhn5\" (UniqueName: \"kubernetes.io/projected/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-kube-api-access-rzhn5\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.794863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.794804 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.795738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.795698 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.797344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.797287 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.818235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.818201 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhn5\" (UniqueName: \"kubernetes.io/projected/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-kube-api-access-rzhn5\") pod \"maas-api-revoke-sub-1ee4c051-zjh99\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.876226 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.876194 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:11.905664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:11.905632 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:12.185003 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:12.184974 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99"] Oct 31 21:45:12.186026 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:12.185996 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43aa5a3a_cd3f_44ae_bd61_911a9c71a54a.slice/crio-bf93302a01f689e9ef48112ab86e49bd81a33a981c7f858eeafbf1829de27d11 WatchSource:0}: Error finding container bf93302a01f689e9ef48112ab86e49bd81a33a981c7f858eeafbf1829de27d11: Status 404 returned error can't find the container with id bf93302a01f689e9ef48112ab86e49bd81a33a981c7f858eeafbf1829de27d11 Oct 31 21:45:12.554588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:12.554556 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" event={"ID":"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a","Type":"ContainerStarted","Data":"c9d83274bb813c5f4e98c1e16fdfaa47b2e5f4aaecd83a0d2552ddac1122b52e"} Oct 31 21:45:12.554742 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:12.554608 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" event={"ID":"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a","Type":"ContainerStarted","Data":"bf93302a01f689e9ef48112ab86e49bd81a33a981c7f858eeafbf1829de27d11"} Oct 31 21:45:12.595492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:12.595445 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" podStartSLOduration=1.595429977 podStartE2EDuration="1.595429977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:12.594579656 +0000 UTC m=+1752.719686899" watchObservedRunningTime="2026-10-31 21:45:12.595429977 +0000 UTC m=+1752.720537243" Oct 31 21:45:12.911472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:12.911438 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:13.563662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:13.563626 4294 generic.go:356] "Generic (PLEG): container finished" podID="43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" containerID="c9d83274bb813c5f4e98c1e16fdfaa47b2e5f4aaecd83a0d2552ddac1122b52e" exitCode=0 Oct 31 21:45:13.563838 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:13.563667 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" event={"ID":"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a","Type":"ContainerDied","Data":"c9d83274bb813c5f4e98c1e16fdfaa47b2e5f4aaecd83a0d2552ddac1122b52e"} Oct 31 21:45:13.564084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:13.564068 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:13.565109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:13.565092 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:45:13.977956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:13.977882 4294 scope.go:122] "RemoveContainer" containerID="d61eb18d74ca2a3f2ff3d5afbac21b422675179eaf3414d39149c1dc39b11c09" Oct 31 21:45:14.572098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.572018 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" event={"ID":"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2","Type":"ContainerStarted","Data":"90b9a03e1ff0a6612aba0928c35832be1a582adb74bc2df18c4aaba526f73626"} Oct 31 21:45:14.740124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.735905 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:14.854152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.854069 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-openshift-ca\") pod \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " Oct 31 21:45:14.854417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.854395 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-xks-ca\") pod \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " Oct 31 21:45:14.854799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.854772 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-openshift-ca" pod "43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" (UID: "43aa5a3a-cd3f-44ae-bd61-911a9c71a54a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:14.854988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.854948 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-kube-api-access-rzhn5\" (UniqueName: \"kubernetes.io/projected/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-kube-api-access-rzhn5\") pod \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\" (UID: \"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a\") " Oct 31 21:45:14.855564 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.855540 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:14.872102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.871902 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-kube-api-access-rzhn5" pod "43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" (UID: "43aa5a3a-cd3f-44ae-bd61-911a9c71a54a"). InnerVolumeSpecName "kube-api-access-rzhn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:14.872102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.871995 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-xks-ca" pod "43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" (UID: "43aa5a3a-cd3f-44ae-bd61-911a9c71a54a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:14.957068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.957043 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzhn5\" (UniqueName: \"kubernetes.io/projected/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-kube-api-access-rzhn5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:14.957068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.957069 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:14.977760 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.977742 4294 scope.go:122] "RemoveContainer" containerID="3686e4f7e23ab426dab20dc6d220c218118a639a2d900b968c532e2558029545" Oct 31 21:45:14.977866 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:14.977852 4294 scope.go:122] "RemoveContainer" containerID="71ee3fff826b55a17ae40f8df197aa6fecb199cc2ce3537b948db7ce80244f0f" Oct 31 21:45:15.581123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.581090 4294 generic.go:356] "Generic (PLEG): container finished" podID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerID="90b9a03e1ff0a6612aba0928c35832be1a582adb74bc2df18c4aaba526f73626" exitCode=60 Oct 31 21:45:15.581569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.581148 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" event={"ID":"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2","Type":"ContainerDied","Data":"90b9a03e1ff0a6612aba0928c35832be1a582adb74bc2df18c4aaba526f73626"} Oct 31 21:45:15.581569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.581204 4294 scope.go:122] "RemoveContainer" containerID="d61eb18d74ca2a3f2ff3d5afbac21b422675179eaf3414d39149c1dc39b11c09" Oct 31 21:45:15.581569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.581534 4294 scope.go:122] "RemoveContainer" containerID="90b9a03e1ff0a6612aba0928c35832be1a582adb74bc2df18c4aaba526f73626" Oct 31 21:45:15.581846 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:15.581820 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5_odh-ai-gateway-infra(d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" Oct 31 21:45:15.583191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.583169 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" event={"ID":"7780c66d-2d0f-4a3d-9131-a41605700f80","Type":"ContainerStarted","Data":"032dc32c25921438b0251605ed02b90a83a3e89caccdf55dcc93d458265e1ded"} Oct 31 21:45:15.584452 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.584436 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" Oct 31 21:45:15.584533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.584474 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99" event={"ID":"43aa5a3a-cd3f-44ae-bd61-911a9c71a54a","Type":"ContainerDied","Data":"bf93302a01f689e9ef48112ab86e49bd81a33a981c7f858eeafbf1829de27d11"} Oct 31 21:45:15.584533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.584518 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf93302a01f689e9ef48112ab86e49bd81a33a981c7f858eeafbf1829de27d11" Oct 31 21:45:15.586291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.586271 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" event={"ID":"2b02ae2b-fddd-4d57-a320-c7e45808ca4f","Type":"ContainerStarted","Data":"9136f710fe0ab32a9d68f064c47d3da11523efe197adefa071ea978db1156a5b"} Oct 31 21:45:15.639283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.639253 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5"] Oct 31 21:45:15.870006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.869972 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw"] Oct 31 21:45:15.870572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.870558 4294 cpu_manager.go:414] "containerMap: removing container" podUID="43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" containerName="revoke-keys" Oct 31 21:45:15.870616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.870573 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" containerName="revoke-keys" Oct 31 21:45:15.870680 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.870671 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" containerName="revoke-keys" Oct 31 21:45:15.874057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.874039 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:15.889099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.889075 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-1f523a\"" Oct 31 21:45:15.889207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.889123 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-1f523a-dockercfg-xqgmd\"" Oct 31 21:45:15.891532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.891504 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw"] Oct 31 21:45:15.971039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.970607 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qzw\" (UniqueName: \"kubernetes.io/projected/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-kube-api-access-m8qzw\") pod \"payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:15.971039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.970749 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:15.974863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.974832 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr"] Oct 31 21:45:15.980203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:15.980177 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.077336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.076326 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plsz\" (UniqueName: \"kubernetes.io/projected/6a940e08-2935-495c-896a-74a979b821c6-kube-api-access-4plsz\") pod \"payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.077336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.076372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a940e08-2935-495c-896a-74a979b821c6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.077336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.076435 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qzw\" (UniqueName: \"kubernetes.io/projected/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-kube-api-access-m8qzw\") pod \"payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:16.077336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.076541 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:16.077676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.077476 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:16.083847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.083819 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr"] Oct 31 21:45:16.101394 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.101327 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qzw\" (UniqueName: \"kubernetes.io/projected/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-kube-api-access-m8qzw\") pod \"payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:16.177927 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.177896 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plsz\" (UniqueName: \"kubernetes.io/projected/6a940e08-2935-495c-896a-74a979b821c6-kube-api-access-4plsz\") pod \"payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.178077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.177932 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a940e08-2935-495c-896a-74a979b821c6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.178692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.178671 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a940e08-2935-495c-896a-74a979b821c6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.201467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.201443 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plsz\" (UniqueName: \"kubernetes.io/projected/6a940e08-2935-495c-896a-74a979b821c6-kube-api-access-4plsz\") pod \"payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.221335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.221313 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:16.383605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.383529 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:16.527778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.527749 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw"] Oct 31 21:45:16.528441 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:16.528419 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90c5062_d9a4_4f71_ae01_ea8f863ef1cd.slice/crio-20b214407eb57a4c3bb14d7b4c3a92afcd8517234d962362e52633aec8ef7237 WatchSource:0}: Error finding container 20b214407eb57a4c3bb14d7b4c3a92afcd8517234d962362e52633aec8ef7237: Status 404 returned error can't find the container with id 20b214407eb57a4c3bb14d7b4c3a92afcd8517234d962362e52633aec8ef7237 Oct 31 21:45:16.595246 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.595221 4294 generic.go:356] "Generic (PLEG): container finished" podID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerID="9136f710fe0ab32a9d68f064c47d3da11523efe197adefa071ea978db1156a5b" exitCode=60 Oct 31 21:45:16.595604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.595294 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" event={"ID":"2b02ae2b-fddd-4d57-a320-c7e45808ca4f","Type":"ContainerDied","Data":"9136f710fe0ab32a9d68f064c47d3da11523efe197adefa071ea978db1156a5b"} Oct 31 21:45:16.595604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.595368 4294 scope.go:122] "RemoveContainer" containerID="3686e4f7e23ab426dab20dc6d220c218118a639a2d900b968c532e2558029545" Oct 31 21:45:16.595793 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.595780 4294 scope.go:122] "RemoveContainer" containerID="9136f710fe0ab32a9d68f064c47d3da11523efe197adefa071ea978db1156a5b" Oct 31 21:45:16.596102 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:16.596080 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx_odh-ai-gateway-infra(2b02ae2b-fddd-4d57-a320-c7e45808ca4f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" Oct 31 21:45:16.599316 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.599282 4294 generic.go:356] "Generic (PLEG): container finished" podID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerID="032dc32c25921438b0251605ed02b90a83a3e89caccdf55dcc93d458265e1ded" exitCode=60 Oct 31 21:45:16.599401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.599339 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" event={"ID":"7780c66d-2d0f-4a3d-9131-a41605700f80","Type":"ContainerDied","Data":"032dc32c25921438b0251605ed02b90a83a3e89caccdf55dcc93d458265e1ded"} Oct 31 21:45:16.599735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.599715 4294 scope.go:122] "RemoveContainer" containerID="032dc32c25921438b0251605ed02b90a83a3e89caccdf55dcc93d458265e1ded" Oct 31 21:45:16.600066 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:16.600039 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j_odh-ai-gateway-infra(7780c66d-2d0f-4a3d-9131-a41605700f80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" Oct 31 21:45:16.601275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.601252 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" event={"ID":"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd","Type":"ContainerStarted","Data":"20b214407eb57a4c3bb14d7b4c3a92afcd8517234d962362e52633aec8ef7237"} Oct 31 21:45:16.614760 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.614741 4294 scope.go:122] "RemoveContainer" containerID="71ee3fff826b55a17ae40f8df197aa6fecb199cc2ce3537b948db7ce80244f0f" Oct 31 21:45:16.686694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.686661 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx"] Oct 31 21:45:16.692813 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:16.692788 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a940e08_2935_495c_896a_74a979b821c6.slice/crio-87ca031fedf8016848724c200294bb336cbce6fba559297b545444a3cd653f53 WatchSource:0}: Error finding container 87ca031fedf8016848724c200294bb336cbce6fba559297b545444a3cd653f53: Status 404 returned error can't find the container with id 87ca031fedf8016848724c200294bb336cbce6fba559297b545444a3cd653f53 Oct 31 21:45:16.698405 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.698379 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr"] Oct 31 21:45:16.737903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.737874 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j"] Oct 31 21:45:16.739611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.739591 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:16.888832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.888799 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-maas-api-service-ca\") pod \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " Oct 31 21:45:16.889053 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.889037 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-kube-api-access-vlskw\" (UniqueName: \"kubernetes.io/projected/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-kube-api-access-vlskw\") pod \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\" (UID: \"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2\") " Oct 31 21:45:16.889153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.889130 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-maas-api-service-ca" pod "d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" (UID: "d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:16.889467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.889449 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:16.891009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.890989 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-kube-api-access-vlskw" pod "d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" (UID: "d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2"). InnerVolumeSpecName "kube-api-access-vlskw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:16.978309 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.978227 4294 scope.go:122] "RemoveContainer" containerID="0f40d4608ec57a4c970184648484757fb3ee94fe9f9556f6bdd0369ce3b7f107" Oct 31 21:45:16.990088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:16.990069 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlskw\" (UniqueName: \"kubernetes.io/projected/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2-kube-api-access-vlskw\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:17.613815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.613485 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" event={"ID":"a577fdb8-0da4-41ae-806f-f3e463e3877e","Type":"ContainerStarted","Data":"a1f072b4bff11fba9bbcd2f8d52fdcbfbab23a926c1638ed19f9d427c3619724"} Oct 31 21:45:17.618710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.618687 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" Oct 31 21:45:17.618836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.618729 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5" event={"ID":"d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2","Type":"ContainerDied","Data":"0c17fa28f8b1f6663c3d77016a4d363b7c528c59b7d47f9b306b75cafd648686"} Oct 31 21:45:17.618836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.618782 4294 scope.go:122] "RemoveContainer" containerID="90b9a03e1ff0a6612aba0928c35832be1a582adb74bc2df18c4aaba526f73626" Oct 31 21:45:17.633620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.633590 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" event={"ID":"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd","Type":"ContainerStarted","Data":"f8606f7ec406c6d062dad19925eeaae6fed2aa38e006978b84b0b262be01e0b6"} Oct 31 21:45:17.633896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.633877 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:17.637340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.635903 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" event={"ID":"6a940e08-2935-495c-896a-74a979b821c6","Type":"ContainerStarted","Data":"19040186a54a7f35cd60a85873533b2500fa2c62f383a581e98618ebe050ab53"} Oct 31 21:45:17.637340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.635952 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" event={"ID":"6a940e08-2935-495c-896a-74a979b821c6","Type":"ContainerStarted","Data":"87ca031fedf8016848724c200294bb336cbce6fba559297b545444a3cd653f53"} Oct 31 21:45:17.637340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.636629 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:17.697380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.694577 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" podStartSLOduration=2.694557589 podStartE2EDuration="2.694557589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:17.693097891 +0000 UTC m=+1757.818205124" watchObservedRunningTime="2026-10-31 21:45:17.694557589 +0000 UTC m=+1757.819664821" Oct 31 21:45:17.732869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.731324 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5"] Oct 31 21:45:17.736786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.736758 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-09f39e-29891385-tlft5"] Oct 31 21:45:17.797119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.797090 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:17.857117 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.857063 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" podStartSLOduration=2.857047945 podStartE2EDuration="2.857047945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:17.85425576 +0000 UTC m=+1757.979362992" watchObservedRunningTime="2026-10-31 21:45:17.857047945 +0000 UTC m=+1757.982155179" Oct 31 21:45:17.905681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.903004 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-maas-api-service-ca\") pod \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " Oct 31 21:45:17.905681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.903077 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-kube-api-access-zvplm\" (UniqueName: \"kubernetes.io/projected/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-kube-api-access-zvplm\") pod \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\" (UID: \"2b02ae2b-fddd-4d57-a320-c7e45808ca4f\") " Oct 31 21:45:17.905681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.904318 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-maas-api-service-ca" pod "2b02ae2b-fddd-4d57-a320-c7e45808ca4f" (UID: "2b02ae2b-fddd-4d57-a320-c7e45808ca4f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:17.906763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.906477 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-kube-api-access-zvplm" pod "2b02ae2b-fddd-4d57-a320-c7e45808ca4f" (UID: "2b02ae2b-fddd-4d57-a320-c7e45808ca4f"). InnerVolumeSpecName "kube-api-access-zvplm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:17.945693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.945667 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:17.982215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.982183 4294 scope.go:122] "RemoveContainer" containerID="de27b9685c9164b6b6826e59dc869f342d96f11b9637fd15e918f2c833da53ed" Oct 31 21:45:17.984084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:17.983207 4294 scope.go:122] "RemoveContainer" containerID="36a83edfa018577fc532a8e3a7b97d123fd0ffbd781f6f15606f69653bc3ca66" Oct 31 21:45:18.009635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.009598 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:18.009635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.009633 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvplm\" (UniqueName: \"kubernetes.io/projected/2b02ae2b-fddd-4d57-a320-c7e45808ca4f-kube-api-access-zvplm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:18.014884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.014854 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" path="/var/lib/kubelet/pods/d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2/volumes" Oct 31 21:45:18.079817 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:18.079786 4294 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b02ae2b_fddd_4d57_a320_c7e45808ca4f.slice/crio-efe55a719386edec2bd22027d1da732f9beb2fe5f7bd866eb024a916b24e6ec1\": RecentStats: unable to find data in memory cache]" Oct 31 21:45:18.110232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.110151 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7780c66d-2d0f-4a3d-9131-a41605700f80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780c66d-2d0f-4a3d-9131-a41605700f80-maas-api-service-ca\") pod \"7780c66d-2d0f-4a3d-9131-a41605700f80\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " Oct 31 21:45:18.110334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.110287 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7780c66d-2d0f-4a3d-9131-a41605700f80-kube-api-access-nn9h4\" (UniqueName: \"kubernetes.io/projected/7780c66d-2d0f-4a3d-9131-a41605700f80-kube-api-access-nn9h4\") pod \"7780c66d-2d0f-4a3d-9131-a41605700f80\" (UID: \"7780c66d-2d0f-4a3d-9131-a41605700f80\") " Oct 31 21:45:18.110555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.110516 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7780c66d-2d0f-4a3d-9131-a41605700f80-maas-api-service-ca" pod "7780c66d-2d0f-4a3d-9131-a41605700f80" (UID: "7780c66d-2d0f-4a3d-9131-a41605700f80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:18.111131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.111103 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7780c66d-2d0f-4a3d-9131-a41605700f80-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:18.112608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.112586 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7780c66d-2d0f-4a3d-9131-a41605700f80-kube-api-access-nn9h4" pod "7780c66d-2d0f-4a3d-9131-a41605700f80" (UID: "7780c66d-2d0f-4a3d-9131-a41605700f80"). InnerVolumeSpecName "kube-api-access-nn9h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:18.212510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.212484 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn9h4\" (UniqueName: \"kubernetes.io/projected/7780c66d-2d0f-4a3d-9131-a41605700f80-kube-api-access-nn9h4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:18.273445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.273406 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h"] Oct 31 21:45:18.274274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274256 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:18.274354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274278 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:18.274354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274317 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274325 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274360 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274367 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274395 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274402 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274414 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274420 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274441 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274448 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274461 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274468 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274480 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274486 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274630 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274644 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274656 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274666 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274674 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" containerName="cleanup" Oct 31 21:45:18.274887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274686 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:18.274887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.274693 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:18.280247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.280227 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.294636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.294607 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h"] Oct 31 21:45:18.295171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.295147 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:45:18.414293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.414257 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.414477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.414414 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.414533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.414480 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2mxh\" (UniqueName: \"kubernetes.io/projected/42805b5f-68dd-460d-a06e-eeae537269e6-kube-api-access-n2mxh\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.515694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.515660 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.515824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.515706 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.515824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.515750 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2mxh\" (UniqueName: \"kubernetes.io/projected/42805b5f-68dd-460d-a06e-eeae537269e6-kube-api-access-n2mxh\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.516313 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.516269 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.517917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.517894 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.549486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.549454 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2mxh\" (UniqueName: \"kubernetes.io/projected/42805b5f-68dd-460d-a06e-eeae537269e6-kube-api-access-n2mxh\") pod \"maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.597365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.597320 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:18.647327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.647274 4294 generic.go:356] "Generic (PLEG): container finished" podID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerID="a1f072b4bff11fba9bbcd2f8d52fdcbfbab23a926c1638ed19f9d427c3619724" exitCode=60 Oct 31 21:45:18.647752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.647358 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" event={"ID":"a577fdb8-0da4-41ae-806f-f3e463e3877e","Type":"ContainerDied","Data":"a1f072b4bff11fba9bbcd2f8d52fdcbfbab23a926c1638ed19f9d427c3619724"} Oct 31 21:45:18.647752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.647443 4294 scope.go:122] "RemoveContainer" containerID="0f40d4608ec57a4c970184648484757fb3ee94fe9f9556f6bdd0369ce3b7f107" Oct 31 21:45:18.647935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.647914 4294 scope.go:122] "RemoveContainer" containerID="a1f072b4bff11fba9bbcd2f8d52fdcbfbab23a926c1638ed19f9d427c3619724" Oct 31 21:45:18.648365 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:18.648291 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv_odh-ai-gateway-infra(a577fdb8-0da4-41ae-806f-f3e463e3877e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" Oct 31 21:45:18.658366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.650661 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" Oct 31 21:45:18.658366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.652224 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j" event={"ID":"7780c66d-2d0f-4a3d-9131-a41605700f80","Type":"ContainerDied","Data":"f749a44d51596da37f0e7cd933482f39e90d8888bc6cbf519d762864baf0f23b"} Oct 31 21:45:18.658366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.654371 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" event={"ID":"5d7cebf9-0de6-4217-9913-4d1aea45ff94","Type":"ContainerStarted","Data":"232079c4dbd8b95fe35ec99fb87e3a0cefb39731c53af381b57d2c7d23470597"} Oct 31 21:45:18.658366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.656609 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" event={"ID":"9a738352-7a54-4d5a-8594-1b960a43705d","Type":"ContainerStarted","Data":"649c32583382be99175671ffacf386173b03f80e3d588f531c68564f38606871"} Oct 31 21:45:18.658984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.658950 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" Oct 31 21:45:18.659465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.659445 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx" event={"ID":"2b02ae2b-fddd-4d57-a320-c7e45808ca4f","Type":"ContainerDied","Data":"efe55a719386edec2bd22027d1da732f9beb2fe5f7bd866eb024a916b24e6ec1"} Oct 31 21:45:18.676662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.676320 4294 scope.go:122] "RemoveContainer" containerID="032dc32c25921438b0251605ed02b90a83a3e89caccdf55dcc93d458265e1ded" Oct 31 21:45:18.700094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.700029 4294 scope.go:122] "RemoveContainer" containerID="9136f710fe0ab32a9d68f064c47d3da11523efe197adefa071ea978db1156a5b" Oct 31 21:45:18.728922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.728847 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx"] Oct 31 21:45:18.733131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.733103 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891385-dhtsx"] Oct 31 21:45:18.804181 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.804151 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j"] Oct 31 21:45:18.809365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.809338 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-80542b-29891385-54t5j"] Oct 31 21:45:18.826250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.826225 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv"] Oct 31 21:45:18.925369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:18.925344 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h"] Oct 31 21:45:18.926372 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:18.926347 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42805b5f_68dd_460d_a06e_eeae537269e6.slice/crio-d8543288bbeaef2de931ddffc37b0208eac51dc0c594a827b19d04776bc23906 WatchSource:0}: Error finding container d8543288bbeaef2de931ddffc37b0208eac51dc0c594a827b19d04776bc23906: Status 404 returned error can't find the container with id d8543288bbeaef2de931ddffc37b0208eac51dc0c594a827b19d04776bc23906 Oct 31 21:45:19.670052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.670018 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" event={"ID":"42805b5f-68dd-460d-a06e-eeae537269e6","Type":"ContainerStarted","Data":"1c032eb93c7d4d145ca6b5e863543dd1cfccda10595a3744da36bd423ee0da86"} Oct 31 21:45:19.670492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.670078 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" event={"ID":"42805b5f-68dd-460d-a06e-eeae537269e6","Type":"ContainerStarted","Data":"d8543288bbeaef2de931ddffc37b0208eac51dc0c594a827b19d04776bc23906"} Oct 31 21:45:19.672889 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.672867 4294 generic.go:356] "Generic (PLEG): container finished" podID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerID="232079c4dbd8b95fe35ec99fb87e3a0cefb39731c53af381b57d2c7d23470597" exitCode=60 Oct 31 21:45:19.673001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.672937 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" event={"ID":"5d7cebf9-0de6-4217-9913-4d1aea45ff94","Type":"ContainerDied","Data":"232079c4dbd8b95fe35ec99fb87e3a0cefb39731c53af381b57d2c7d23470597"} Oct 31 21:45:19.673001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.672996 4294 scope.go:122] "RemoveContainer" containerID="de27b9685c9164b6b6826e59dc869f342d96f11b9637fd15e918f2c833da53ed" Oct 31 21:45:19.673356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.673338 4294 scope.go:122] "RemoveContainer" containerID="232079c4dbd8b95fe35ec99fb87e3a0cefb39731c53af381b57d2c7d23470597" Oct 31 21:45:19.673679 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:19.673651 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891385-j4bgh_odh-ai-gateway-infra(5d7cebf9-0de6-4217-9913-4d1aea45ff94)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" Oct 31 21:45:19.675196 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.675172 4294 generic.go:356] "Generic (PLEG): container finished" podID="9a738352-7a54-4d5a-8594-1b960a43705d" containerID="649c32583382be99175671ffacf386173b03f80e3d588f531c68564f38606871" exitCode=60 Oct 31 21:45:19.675311 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.675220 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" event={"ID":"9a738352-7a54-4d5a-8594-1b960a43705d","Type":"ContainerDied","Data":"649c32583382be99175671ffacf386173b03f80e3d588f531c68564f38606871"} Oct 31 21:45:19.675604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.675587 4294 scope.go:122] "RemoveContainer" containerID="649c32583382be99175671ffacf386173b03f80e3d588f531c68564f38606871" Oct 31 21:45:19.675911 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:19.675888 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz_odh-ai-gateway-infra(9a738352-7a54-4d5a-8594-1b960a43705d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" Oct 31 21:45:19.685978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.685959 4294 scope.go:122] "RemoveContainer" containerID="36a83edfa018577fc532a8e3a7b97d123fd0ffbd781f6f15606f69653bc3ca66" Oct 31 21:45:19.751319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.751264 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" podStartSLOduration=1.751252144 podStartE2EDuration="1.751252144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:19.749514412 +0000 UTC m=+1759.874621642" watchObservedRunningTime="2026-10-31 21:45:19.751252144 +0000 UTC m=+1759.876359375" Oct 31 21:45:19.782094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.782062 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh"] Oct 31 21:45:19.860474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.860441 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz"] Oct 31 21:45:19.881232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.881206 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:19.985682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.985651 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" path="/var/lib/kubelet/pods/2b02ae2b-fddd-4d57-a320-c7e45808ca4f/volumes" Oct 31 21:45:19.986118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:19.986105 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7780c66d-2d0f-4a3d-9131-a41605700f80" path="/var/lib/kubelet/pods/7780c66d-2d0f-4a3d-9131-a41605700f80/volumes" Oct 31 21:45:20.040645 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.040612 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a577fdb8-0da4-41ae-806f-f3e463e3877e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a577fdb8-0da4-41ae-806f-f3e463e3877e-maas-api-service-ca\") pod \"a577fdb8-0da4-41ae-806f-f3e463e3877e\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " Oct 31 21:45:20.040806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.040657 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a577fdb8-0da4-41ae-806f-f3e463e3877e-kube-api-access-pmrlb\" (UniqueName: \"kubernetes.io/projected/a577fdb8-0da4-41ae-806f-f3e463e3877e-kube-api-access-pmrlb\") pod \"a577fdb8-0da4-41ae-806f-f3e463e3877e\" (UID: \"a577fdb8-0da4-41ae-806f-f3e463e3877e\") " Oct 31 21:45:20.041050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.041023 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a577fdb8-0da4-41ae-806f-f3e463e3877e-maas-api-service-ca" pod "a577fdb8-0da4-41ae-806f-f3e463e3877e" (UID: "a577fdb8-0da4-41ae-806f-f3e463e3877e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:20.041147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.041131 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a577fdb8-0da4-41ae-806f-f3e463e3877e-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:20.042840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.042817 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a577fdb8-0da4-41ae-806f-f3e463e3877e-kube-api-access-pmrlb" pod "a577fdb8-0da4-41ae-806f-f3e463e3877e" (UID: "a577fdb8-0da4-41ae-806f-f3e463e3877e"). InnerVolumeSpecName "kube-api-access-pmrlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:20.142171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.142126 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrlb\" (UniqueName: \"kubernetes.io/projected/a577fdb8-0da4-41ae-806f-f3e463e3877e-kube-api-access-pmrlb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:20.683125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.683085 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" event={"ID":"a577fdb8-0da4-41ae-806f-f3e463e3877e","Type":"ContainerDied","Data":"4f7553b9d7e94fc54b51313477d6bc24d8412df457f30974559bf104a23f3779"} Oct 31 21:45:20.683597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.683159 4294 scope.go:122] "RemoveContainer" containerID="a1f072b4bff11fba9bbcd2f8d52fdcbfbab23a926c1638ed19f9d427c3619724" Oct 31 21:45:20.683597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.683093 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv" Oct 31 21:45:20.684849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.684828 4294 generic.go:356] "Generic (PLEG): container finished" podID="42805b5f-68dd-460d-a06e-eeae537269e6" containerID="1c032eb93c7d4d145ca6b5e863543dd1cfccda10595a3744da36bd423ee0da86" exitCode=0 Oct 31 21:45:20.684975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.684915 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" event={"ID":"42805b5f-68dd-460d-a06e-eeae537269e6","Type":"ContainerDied","Data":"1c032eb93c7d4d145ca6b5e863543dd1cfccda10595a3744da36bd423ee0da86"} Oct 31 21:45:20.863276 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.863248 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:20.877132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.877102 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv"] Oct 31 21:45:20.880554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.880530 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-9c8b33-29891385-jmgkv"] Oct 31 21:45:20.892559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.892539 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:20.955557 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.955470 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d7cebf9-0de6-4217-9913-4d1aea45ff94-kube-api-access-v4fqc\" (UniqueName: \"kubernetes.io/projected/5d7cebf9-0de6-4217-9913-4d1aea45ff94-kube-api-access-v4fqc\") pod \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " Oct 31 21:45:20.955707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.955566 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d7cebf9-0de6-4217-9913-4d1aea45ff94-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7cebf9-0de6-4217-9913-4d1aea45ff94-maas-api-service-ca\") pod \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\" (UID: \"5d7cebf9-0de6-4217-9913-4d1aea45ff94\") " Oct 31 21:45:20.956663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.956627 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7cebf9-0de6-4217-9913-4d1aea45ff94-maas-api-service-ca" pod "5d7cebf9-0de6-4217-9913-4d1aea45ff94" (UID: "5d7cebf9-0de6-4217-9913-4d1aea45ff94"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:20.961403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:20.961360 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7cebf9-0de6-4217-9913-4d1aea45ff94-kube-api-access-v4fqc" pod "5d7cebf9-0de6-4217-9913-4d1aea45ff94" (UID: "5d7cebf9-0de6-4217-9913-4d1aea45ff94"). InnerVolumeSpecName "kube-api-access-v4fqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:21.056835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.056805 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a738352-7a54-4d5a-8594-1b960a43705d-kube-api-access-jn94t\" (UniqueName: \"kubernetes.io/projected/9a738352-7a54-4d5a-8594-1b960a43705d-kube-api-access-jn94t\") pod \"9a738352-7a54-4d5a-8594-1b960a43705d\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " Oct 31 21:45:21.056983 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.056958 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a738352-7a54-4d5a-8594-1b960a43705d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a738352-7a54-4d5a-8594-1b960a43705d-maas-api-service-ca\") pod \"9a738352-7a54-4d5a-8594-1b960a43705d\" (UID: \"9a738352-7a54-4d5a-8594-1b960a43705d\") " Oct 31 21:45:21.057348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.057295 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a738352-7a54-4d5a-8594-1b960a43705d-maas-api-service-ca" pod "9a738352-7a54-4d5a-8594-1b960a43705d" (UID: "9a738352-7a54-4d5a-8594-1b960a43705d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:21.057444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.057425 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a738352-7a54-4d5a-8594-1b960a43705d-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.057519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.057454 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fqc\" (UniqueName: \"kubernetes.io/projected/5d7cebf9-0de6-4217-9913-4d1aea45ff94-kube-api-access-v4fqc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.057519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.057470 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d7cebf9-0de6-4217-9913-4d1aea45ff94-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.058962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.058939 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a738352-7a54-4d5a-8594-1b960a43705d-kube-api-access-jn94t" pod "9a738352-7a54-4d5a-8594-1b960a43705d" (UID: "9a738352-7a54-4d5a-8594-1b960a43705d"). InnerVolumeSpecName "kube-api-access-jn94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:21.159134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.159099 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn94t\" (UniqueName: \"kubernetes.io/projected/9a738352-7a54-4d5a-8594-1b960a43705d-kube-api-access-jn94t\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.697749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.697722 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" Oct 31 21:45:21.698237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.697726 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh" event={"ID":"5d7cebf9-0de6-4217-9913-4d1aea45ff94","Type":"ContainerDied","Data":"95affb2253251774d9d39ecba795efdf6625fb91795470be89acb40154cc2c8b"} Oct 31 21:45:21.698237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.697863 4294 scope.go:122] "RemoveContainer" containerID="232079c4dbd8b95fe35ec99fb87e3a0cefb39731c53af381b57d2c7d23470597" Oct 31 21:45:21.701396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.701377 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" Oct 31 21:45:21.701504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.701423 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz" event={"ID":"9a738352-7a54-4d5a-8594-1b960a43705d","Type":"ContainerDied","Data":"73b1e5edaad0a3515046582d6845a7cfa809239d5e12e4b21b6299fb6f56b6e7"} Oct 31 21:45:21.713089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.713065 4294 scope.go:122] "RemoveContainer" containerID="649c32583382be99175671ffacf386173b03f80e3d588f531c68564f38606871" Oct 31 21:45:21.750632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.750605 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh"] Oct 31 21:45:21.755640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.755612 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891385-j4bgh"] Oct 31 21:45:21.807596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.807565 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz"] Oct 31 21:45:21.819431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.819401 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891385-vnjrz"] Oct 31 21:45:21.861873 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.861850 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:21.968138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.968064 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-xks-ca\") pod \"42805b5f-68dd-460d-a06e-eeae537269e6\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " Oct 31 21:45:21.968138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.968116 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42805b5f-68dd-460d-a06e-eeae537269e6-kube-api-access-n2mxh\" (UniqueName: \"kubernetes.io/projected/42805b5f-68dd-460d-a06e-eeae537269e6-kube-api-access-n2mxh\") pod \"42805b5f-68dd-460d-a06e-eeae537269e6\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " Oct 31 21:45:21.968375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.968165 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-openshift-ca\") pod \"42805b5f-68dd-460d-a06e-eeae537269e6\" (UID: \"42805b5f-68dd-460d-a06e-eeae537269e6\") " Oct 31 21:45:21.968578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.968548 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-openshift-ca" pod "42805b5f-68dd-460d-a06e-eeae537269e6" (UID: "42805b5f-68dd-460d-a06e-eeae537269e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:21.968684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.968648 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:21.970222 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.970199 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42805b5f-68dd-460d-a06e-eeae537269e6-kube-api-access-n2mxh" pod "42805b5f-68dd-460d-a06e-eeae537269e6" (UID: "42805b5f-68dd-460d-a06e-eeae537269e6"). InnerVolumeSpecName "kube-api-access-n2mxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:21.970517 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.970502 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-xks-ca" pod "42805b5f-68dd-460d-a06e-eeae537269e6" (UID: "42805b5f-68dd-460d-a06e-eeae537269e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:21.985471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.985445 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" path="/var/lib/kubelet/pods/5d7cebf9-0de6-4217-9913-4d1aea45ff94/volumes" Oct 31 21:45:21.985858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.985846 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" path="/var/lib/kubelet/pods/9a738352-7a54-4d5a-8594-1b960a43705d/volumes" Oct 31 21:45:21.986208 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:21.986196 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" path="/var/lib/kubelet/pods/a577fdb8-0da4-41ae-806f-f3e463e3877e/volumes" Oct 31 21:45:22.070152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:22.070122 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42805b5f-68dd-460d-a06e-eeae537269e6-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:22.070152 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:22.070149 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2mxh\" (UniqueName: \"kubernetes.io/projected/42805b5f-68dd-460d-a06e-eeae537269e6-kube-api-access-n2mxh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:22.709362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:22.709328 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" Oct 31 21:45:22.709362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:22.709341 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h" event={"ID":"42805b5f-68dd-460d-a06e-eeae537269e6","Type":"ContainerDied","Data":"d8543288bbeaef2de931ddffc37b0208eac51dc0c594a827b19d04776bc23906"} Oct 31 21:45:22.709872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:22.709412 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8543288bbeaef2de931ddffc37b0208eac51dc0c594a827b19d04776bc23906" Oct 31 21:45:24.191632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.191589 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b"] Oct 31 21:45:24.192399 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192376 4294 cpu_manager.go:414] "containerMap: removing container" podUID="42805b5f-68dd-460d-a06e-eeae537269e6" containerName="revoke-keys" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192401 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42805b5f-68dd-460d-a06e-eeae537269e6" containerName="revoke-keys" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192427 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192435 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192460 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192468 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192478 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192485 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192496 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.192516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192503 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192532 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192539 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192556 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192563 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192580 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192586 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192598 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192605 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192616 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192624 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192768 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:24.192785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192783 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192791 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192803 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b02ae2b-fddd-4d57-a320-c7e45808ca4f" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192813 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192825 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d7cebf9-0de6-4217-9913-4d1aea45ff94" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192836 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192844 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a738352-7a54-4d5a-8594-1b960a43705d" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192855 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192864 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d494a4-63cc-4d4d-9fc0-6c00bec5ece2" containerName="cleanup" Oct 31 21:45:24.193103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.192874 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="42805b5f-68dd-460d-a06e-eeae537269e6" containerName="revoke-keys" Oct 31 21:45:24.196783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.196761 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.211992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.211960 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b"] Oct 31 21:45:24.220043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.219404 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:45:24.220043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.219441 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:45:24.307325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.306059 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdbvk\" (UniqueName: \"kubernetes.io/projected/1179a0fb-d792-4464-94fd-c894b3dd3b1c-kube-api-access-qdbvk\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.307325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.306124 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.307325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.306205 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.412330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.411016 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.412330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.411130 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.412330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.411379 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdbvk\" (UniqueName: \"kubernetes.io/projected/1179a0fb-d792-4464-94fd-c894b3dd3b1c-kube-api-access-qdbvk\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.412604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.412451 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.449376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.435326 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.449376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.438970 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdbvk\" (UniqueName: \"kubernetes.io/projected/1179a0fb-d792-4464-94fd-c894b3dd3b1c-kube-api-access-qdbvk\") pod \"maas-api-revoke-sub-dc17ebf4-lzh7b\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.514323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.513378 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:24.942617 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:24.942278 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b"] Oct 31 21:45:25.780438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:25.780400 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" event={"ID":"1179a0fb-d792-4464-94fd-c894b3dd3b1c","Type":"ContainerStarted","Data":"e3486b89192d9689869b61f50859921d36cd0d18ca6ed7c33834adc085bd7942"} Oct 31 21:45:25.781029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:25.781003 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" event={"ID":"1179a0fb-d792-4464-94fd-c894b3dd3b1c","Type":"ContainerStarted","Data":"f0bba3c7f23bd143d32d50dd376ffe93ead3adf3349c32ff78569ebc8ac17bad"} Oct 31 21:45:25.852235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:25.852201 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4"] Oct 31 21:45:25.857053 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:25.857012 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f4569e-mhpf4"] Oct 31 21:45:25.858099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:25.858050 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" podStartSLOduration=1.8580357699999999 podStartE2EDuration="1.85803577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:25.845356058 +0000 UTC m=+1765.970463290" watchObservedRunningTime="2026-10-31 21:45:25.85803577 +0000 UTC m=+1765.983143000" Oct 31 21:45:26.002506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:26.002467 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a903c3-b760-418b-a5b2-8b3df60ff42c" path="/var/lib/kubelet/pods/88a903c3-b760-418b-a5b2-8b3df60ff42c/volumes" Oct 31 21:45:26.787963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:26.787930 4294 generic.go:356] "Generic (PLEG): container finished" podID="1179a0fb-d792-4464-94fd-c894b3dd3b1c" containerID="e3486b89192d9689869b61f50859921d36cd0d18ca6ed7c33834adc085bd7942" exitCode=0 Oct 31 21:45:26.788419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:26.787996 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" event={"ID":"1179a0fb-d792-4464-94fd-c894b3dd3b1c","Type":"ContainerDied","Data":"e3486b89192d9689869b61f50859921d36cd0d18ca6ed7c33834adc085bd7942"} Oct 31 21:45:27.996003 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:27.995975 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:28.146741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.146647 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-openshift-ca\") pod \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " Oct 31 21:45:28.146898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.146797 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1179a0fb-d792-4464-94fd-c894b3dd3b1c-kube-api-access-qdbvk\" (UniqueName: \"kubernetes.io/projected/1179a0fb-d792-4464-94fd-c894b3dd3b1c-kube-api-access-qdbvk\") pod \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " Oct 31 21:45:28.146966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.146943 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-xks-ca\") pod \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\" (UID: \"1179a0fb-d792-4464-94fd-c894b3dd3b1c\") " Oct 31 21:45:28.147095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.147068 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-openshift-ca" pod "1179a0fb-d792-4464-94fd-c894b3dd3b1c" (UID: "1179a0fb-d792-4464-94fd-c894b3dd3b1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:28.147569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.147546 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:28.149135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.149113 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1179a0fb-d792-4464-94fd-c894b3dd3b1c-kube-api-access-qdbvk" pod "1179a0fb-d792-4464-94fd-c894b3dd3b1c" (UID: "1179a0fb-d792-4464-94fd-c894b3dd3b1c"). InnerVolumeSpecName "kube-api-access-qdbvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:28.149337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.149285 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-xks-ca" pod "1179a0fb-d792-4464-94fd-c894b3dd3b1c" (UID: "1179a0fb-d792-4464-94fd-c894b3dd3b1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:28.248828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.248803 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdbvk\" (UniqueName: \"kubernetes.io/projected/1179a0fb-d792-4464-94fd-c894b3dd3b1c-kube-api-access-qdbvk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:28.248828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.248826 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1179a0fb-d792-4464-94fd-c894b3dd3b1c-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:28.663175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.663148 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:45:28.663353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.663219 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:45:28.805332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.805293 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" Oct 31 21:45:28.805492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.805336 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b" event={"ID":"1179a0fb-d792-4464-94fd-c894b3dd3b1c","Type":"ContainerDied","Data":"f0bba3c7f23bd143d32d50dd376ffe93ead3adf3349c32ff78569ebc8ac17bad"} Oct 31 21:45:28.805492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:28.805386 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0bba3c7f23bd143d32d50dd376ffe93ead3adf3349c32ff78569ebc8ac17bad" Oct 31 21:45:31.312400 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:45:31.312366 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:45:31.742383 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:31.742348 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h"] Oct 31 21:45:31.753848 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:31.753820 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d43b66-x6z8h"] Oct 31 21:45:31.984550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:31.984519 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42805b5f-68dd-460d-a06e-eeae537269e6" path="/var/lib/kubelet/pods/42805b5f-68dd-460d-a06e-eeae537269e6/volumes" Oct 31 21:45:36.208973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.208940 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5"] Oct 31 21:45:36.209486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.209472 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:36.209536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.209486 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:36.209536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.209515 4294 cpu_manager.go:414] "containerMap: removing container" podUID="1179a0fb-d792-4464-94fd-c894b3dd3b1c" containerName="revoke-keys" Oct 31 21:45:36.209536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.209520 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1179a0fb-d792-4464-94fd-c894b3dd3b1c" containerName="revoke-keys" Oct 31 21:45:36.209640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.209607 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1179a0fb-d792-4464-94fd-c894b3dd3b1c" containerName="revoke-keys" Oct 31 21:45:36.209640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.209619 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a577fdb8-0da4-41ae-806f-f3e463e3877e" containerName="cleanup" Oct 31 21:45:36.214433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.214416 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.232294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.232261 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5"] Oct 31 21:45:36.239674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.239652 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:45:36.239807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.239707 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:45:36.299186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.299157 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.299340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.299219 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.299386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.299357 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jpl\" (UniqueName: \"kubernetes.io/projected/231931ab-30d5-4b29-ab33-a9b5ffa14290-kube-api-access-b9jpl\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.400606 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.400571 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.400791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.400660 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.400791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.400716 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jpl\" (UniqueName: \"kubernetes.io/projected/231931ab-30d5-4b29-ab33-a9b5ffa14290-kube-api-access-b9jpl\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.401284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.401251 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.402993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.402968 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.424093 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.424061 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jpl\" (UniqueName: \"kubernetes.io/projected/231931ab-30d5-4b29-ab33-a9b5ffa14290-kube-api-access-b9jpl\") pod \"maas-api-revoke-sub-dc9421a5-x9mn5\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.527295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.527265 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:36.828614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.828586 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5"] Oct 31 21:45:36.830647 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:36.830613 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231931ab_30d5_4b29_ab33_a9b5ffa14290.slice/crio-63afa772cf40958835d356d01a37595ac2cb4816801d56e9486363a0eef1c528 WatchSource:0}: Error finding container 63afa772cf40958835d356d01a37595ac2cb4816801d56e9486363a0eef1c528: Status 404 returned error can't find the container with id 63afa772cf40958835d356d01a37595ac2cb4816801d56e9486363a0eef1c528 Oct 31 21:45:36.858263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:36.858228 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" event={"ID":"231931ab-30d5-4b29-ab33-a9b5ffa14290","Type":"ContainerStarted","Data":"63afa772cf40958835d356d01a37595ac2cb4816801d56e9486363a0eef1c528"} Oct 31 21:45:37.414962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.414887 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d"] Oct 31 21:45:37.418927 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.418911 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.439846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.439814 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d"] Oct 31 21:45:37.516295 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.516248 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.516486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.516450 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.516538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.516487 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmt4n\" (UniqueName: \"kubernetes.io/projected/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-kube-api-access-zmt4n\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.617994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.617957 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.618187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.618028 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmt4n\" (UniqueName: \"kubernetes.io/projected/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-kube-api-access-zmt4n\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.618187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.618145 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.618785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.618764 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.620480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.620454 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.640967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.640942 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmt4n\" (UniqueName: \"kubernetes.io/projected/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-kube-api-access-zmt4n\") pod \"maas-api-revoke-sub-2c22e879-bdd7d\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.732644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.732621 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:37.867141 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.867104 4294 generic.go:356] "Generic (PLEG): container finished" podID="231931ab-30d5-4b29-ab33-a9b5ffa14290" containerID="929962e354f44eddcf12b7c220b4010ff9156ae5365d1f1c9d204673ee548570" exitCode=0 Oct 31 21:45:37.867327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:37.867191 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" event={"ID":"231931ab-30d5-4b29-ab33-a9b5ffa14290","Type":"ContainerDied","Data":"929962e354f44eddcf12b7c220b4010ff9156ae5365d1f1c9d204673ee548570"} Oct 31 21:45:38.037780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:38.037749 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d"] Oct 31 21:45:38.038795 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:38.038764 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec89a43_ef16_48e5_8d8c_01b8628b7e1f.slice/crio-86033c39a9bc0083f3cb8672a77140df13f2a7e61b7e440487032506f2f4351c WatchSource:0}: Error finding container 86033c39a9bc0083f3cb8672a77140df13f2a7e61b7e440487032506f2f4351c: Status 404 returned error can't find the container with id 86033c39a9bc0083f3cb8672a77140df13f2a7e61b7e440487032506f2f4351c Oct 31 21:45:38.875806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:38.875767 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" event={"ID":"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f","Type":"ContainerStarted","Data":"4d76bdad79d6b517eca45c0abfc09260164830ba96f2aabed16d942371184d53"} Oct 31 21:45:38.876232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:38.875847 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" event={"ID":"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f","Type":"ContainerStarted","Data":"86033c39a9bc0083f3cb8672a77140df13f2a7e61b7e440487032506f2f4351c"} Oct 31 21:45:38.916821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:38.916770 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" podStartSLOduration=1.916755779 podStartE2EDuration="1.916755779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:38.915680229 +0000 UTC m=+1779.040787459" watchObservedRunningTime="2026-10-31 21:45:38.916755779 +0000 UTC m=+1779.041863010" Oct 31 21:45:39.069235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.069209 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:39.138853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.138775 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-openshift-ca\") pod \"231931ab-30d5-4b29-ab33-a9b5ffa14290\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " Oct 31 21:45:39.139021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.138956 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231931ab-30d5-4b29-ab33-a9b5ffa14290-kube-api-access-b9jpl\" (UniqueName: \"kubernetes.io/projected/231931ab-30d5-4b29-ab33-a9b5ffa14290-kube-api-access-b9jpl\") pod \"231931ab-30d5-4b29-ab33-a9b5ffa14290\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " Oct 31 21:45:39.139021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.139004 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-xks-ca\") pod \"231931ab-30d5-4b29-ab33-a9b5ffa14290\" (UID: \"231931ab-30d5-4b29-ab33-a9b5ffa14290\") " Oct 31 21:45:39.139476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.139142 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-openshift-ca" pod "231931ab-30d5-4b29-ab33-a9b5ffa14290" (UID: "231931ab-30d5-4b29-ab33-a9b5ffa14290"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:39.139619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.139597 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:39.141387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.141361 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-xks-ca" pod "231931ab-30d5-4b29-ab33-a9b5ffa14290" (UID: "231931ab-30d5-4b29-ab33-a9b5ffa14290"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:39.141685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.141669 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231931ab-30d5-4b29-ab33-a9b5ffa14290-kube-api-access-b9jpl" pod "231931ab-30d5-4b29-ab33-a9b5ffa14290" (UID: "231931ab-30d5-4b29-ab33-a9b5ffa14290"). InnerVolumeSpecName "kube-api-access-b9jpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:39.240615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.240583 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231931ab-30d5-4b29-ab33-a9b5ffa14290-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:39.240615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.240612 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jpl\" (UniqueName: \"kubernetes.io/projected/231931ab-30d5-4b29-ab33-a9b5ffa14290-kube-api-access-b9jpl\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:39.887995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.887961 4294 generic.go:356] "Generic (PLEG): container finished" podID="7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" containerID="4d76bdad79d6b517eca45c0abfc09260164830ba96f2aabed16d942371184d53" exitCode=0 Oct 31 21:45:39.888468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.888044 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" event={"ID":"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f","Type":"ContainerDied","Data":"4d76bdad79d6b517eca45c0abfc09260164830ba96f2aabed16d942371184d53"} Oct 31 21:45:39.889422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.889401 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" event={"ID":"231931ab-30d5-4b29-ab33-a9b5ffa14290","Type":"ContainerDied","Data":"63afa772cf40958835d356d01a37595ac2cb4816801d56e9486363a0eef1c528"} Oct 31 21:45:39.889422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.889419 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5" Oct 31 21:45:39.889597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:39.889435 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63afa772cf40958835d356d01a37595ac2cb4816801d56e9486363a0eef1c528" Oct 31 21:45:40.074115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.074083 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp"] Oct 31 21:45:40.074664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.074650 4294 cpu_manager.go:414] "containerMap: removing container" podUID="231931ab-30d5-4b29-ab33-a9b5ffa14290" containerName="revoke-keys" Oct 31 21:45:40.074717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.074666 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231931ab-30d5-4b29-ab33-a9b5ffa14290" containerName="revoke-keys" Oct 31 21:45:40.074775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.074766 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="231931ab-30d5-4b29-ab33-a9b5ffa14290" containerName="revoke-keys" Oct 31 21:45:40.078168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.078152 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.098338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.098089 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp"] Oct 31 21:45:40.155461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.155390 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x4s7\" (UniqueName: \"kubernetes.io/projected/c776ea10-f633-4145-8801-970826053d31-kube-api-access-4x4s7\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.155461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.155424 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c776ea10-f633-4145-8801-970826053d31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.155657 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.155499 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c776ea10-f633-4145-8801-970826053d31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.256511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.256478 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c776ea10-f633-4145-8801-970826053d31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.256694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.256677 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x4s7\" (UniqueName: \"kubernetes.io/projected/c776ea10-f633-4145-8801-970826053d31-kube-api-access-4x4s7\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.256740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.256707 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c776ea10-f633-4145-8801-970826053d31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.257380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.257342 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c776ea10-f633-4145-8801-970826053d31-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.258718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.258694 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c776ea10-f633-4145-8801-970826053d31-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.282883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.282855 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x4s7\" (UniqueName: \"kubernetes.io/projected/c776ea10-f633-4145-8801-970826053d31-kube-api-access-4x4s7\") pod \"maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.393193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.393161 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:40.735129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.735012 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp"] Oct 31 21:45:40.738011 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:40.737973 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc776ea10_f633_4145_8801_970826053d31.slice/crio-c42aae11afdcc65d88a449cf2470896ed630731dc3106d10c71b8bcf8bcfa989 WatchSource:0}: Error finding container c42aae11afdcc65d88a449cf2470896ed630731dc3106d10c71b8bcf8bcfa989: Status 404 returned error can't find the container with id c42aae11afdcc65d88a449cf2470896ed630731dc3106d10c71b8bcf8bcfa989 Oct 31 21:45:40.899945 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:40.899909 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" event={"ID":"c776ea10-f633-4145-8801-970826053d31","Type":"ContainerStarted","Data":"c42aae11afdcc65d88a449cf2470896ed630731dc3106d10c71b8bcf8bcfa989"} Oct 31 21:45:41.077441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.077408 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:41.173187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.173157 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-xks-ca\") pod \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " Oct 31 21:45:41.173654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.173633 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-openshift-ca\") pod \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " Oct 31 21:45:41.173991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.173835 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-kube-api-access-zmt4n\" (UniqueName: \"kubernetes.io/projected/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-kube-api-access-zmt4n\") pod \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\" (UID: \"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f\") " Oct 31 21:45:41.174618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.174215 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-openshift-ca" pod "7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" (UID: "7ec89a43-ef16-48e5-8d8c-01b8628b7e1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:41.174618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.174585 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:41.184626 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.184596 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-xks-ca" pod "7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" (UID: "7ec89a43-ef16-48e5-8d8c-01b8628b7e1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:41.186196 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.186172 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-kube-api-access-zmt4n" pod "7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" (UID: "7ec89a43-ef16-48e5-8d8c-01b8628b7e1f"). InnerVolumeSpecName "kube-api-access-zmt4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:41.276380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.276280 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:41.276380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.276325 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmt4n\" (UniqueName: \"kubernetes.io/projected/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f-kube-api-access-zmt4n\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:41.919402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.919350 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" event={"ID":"c776ea10-f633-4145-8801-970826053d31","Type":"ContainerStarted","Data":"e8c36a4bb16599e21278a094c77038f4ba27298a81ad41123d50cf6424b5d23d"} Oct 31 21:45:41.924169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.924134 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" event={"ID":"7ec89a43-ef16-48e5-8d8c-01b8628b7e1f","Type":"ContainerDied","Data":"86033c39a9bc0083f3cb8672a77140df13f2a7e61b7e440487032506f2f4351c"} Oct 31 21:45:41.924360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.924186 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86033c39a9bc0083f3cb8672a77140df13f2a7e61b7e440487032506f2f4351c" Oct 31 21:45:41.924360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.924242 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d" Oct 31 21:45:41.968155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:41.968115 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" podStartSLOduration=1.968102951 podStartE2EDuration="1.968102951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:41.961158119 +0000 UTC m=+1782.086265352" watchObservedRunningTime="2026-10-31 21:45:41.968102951 +0000 UTC m=+1782.093210181" Oct 31 21:45:42.931968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:42.931934 4294 generic.go:356] "Generic (PLEG): container finished" podID="c776ea10-f633-4145-8801-970826053d31" containerID="e8c36a4bb16599e21278a094c77038f4ba27298a81ad41123d50cf6424b5d23d" exitCode=0 Oct 31 21:45:42.932395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:42.931983 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" event={"ID":"c776ea10-f633-4145-8801-970826053d31","Type":"ContainerDied","Data":"e8c36a4bb16599e21278a094c77038f4ba27298a81ad41123d50cf6424b5d23d"} Oct 31 21:45:44.085874 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.085846 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:44.219835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.219755 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c776ea10-f633-4145-8801-970826053d31-kube-api-access-4x4s7\" (UniqueName: \"kubernetes.io/projected/c776ea10-f633-4145-8801-970826053d31-kube-api-access-4x4s7\") pod \"c776ea10-f633-4145-8801-970826053d31\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " Oct 31 21:45:44.219977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.219884 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c776ea10-f633-4145-8801-970826053d31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c776ea10-f633-4145-8801-970826053d31-maas-api-xks-ca\") pod \"c776ea10-f633-4145-8801-970826053d31\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " Oct 31 21:45:44.220044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.220028 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c776ea10-f633-4145-8801-970826053d31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c776ea10-f633-4145-8801-970826053d31-maas-api-openshift-ca\") pod \"c776ea10-f633-4145-8801-970826053d31\" (UID: \"c776ea10-f633-4145-8801-970826053d31\") " Oct 31 21:45:44.220433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.220402 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c776ea10-f633-4145-8801-970826053d31-maas-api-openshift-ca" pod "c776ea10-f633-4145-8801-970826053d31" (UID: "c776ea10-f633-4145-8801-970826053d31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:44.220648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.220625 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c776ea10-f633-4145-8801-970826053d31-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:44.221884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.221855 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c776ea10-f633-4145-8801-970826053d31-maas-api-xks-ca" pod "c776ea10-f633-4145-8801-970826053d31" (UID: "c776ea10-f633-4145-8801-970826053d31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:44.221969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.221954 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c776ea10-f633-4145-8801-970826053d31-kube-api-access-4x4s7" pod "c776ea10-f633-4145-8801-970826053d31" (UID: "c776ea10-f633-4145-8801-970826053d31"). InnerVolumeSpecName "kube-api-access-4x4s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:44.322033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.321999 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x4s7\" (UniqueName: \"kubernetes.io/projected/c776ea10-f633-4145-8801-970826053d31-kube-api-access-4x4s7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:44.322033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.322035 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c776ea10-f633-4145-8801-970826053d31-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:44.946114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.946085 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" Oct 31 21:45:44.946287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.946115 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp" event={"ID":"c776ea10-f633-4145-8801-970826053d31","Type":"ContainerDied","Data":"c42aae11afdcc65d88a449cf2470896ed630731dc3106d10c71b8bcf8bcfa989"} Oct 31 21:45:44.946287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:44.946159 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c42aae11afdcc65d88a449cf2470896ed630731dc3106d10c71b8bcf8bcfa989" Oct 31 21:45:46.562228 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.562194 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz"] Oct 31 21:45:46.562795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.562774 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" containerName="revoke-keys" Oct 31 21:45:46.562795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.562791 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" containerName="revoke-keys" Oct 31 21:45:46.562941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.562803 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c776ea10-f633-4145-8801-970826053d31" containerName="revoke-keys" Oct 31 21:45:46.562941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.562808 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c776ea10-f633-4145-8801-970826053d31" containerName="revoke-keys" Oct 31 21:45:46.562941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.562914 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" containerName="revoke-keys" Oct 31 21:45:46.562941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.562922 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c776ea10-f633-4145-8801-970826053d31" containerName="revoke-keys" Oct 31 21:45:46.566773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.566756 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.584351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.584319 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz"] Oct 31 21:45:46.594629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.593079 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:45:46.597524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.596123 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:45:46.597732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.597702 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl"] Oct 31 21:45:46.603092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.603074 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.620574 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.620549 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl"] Oct 31 21:45:46.655707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.655672 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.655969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.655944 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.656100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.656068 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64flk\" (UniqueName: \"kubernetes.io/projected/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-kube-api-access-64flk\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.757861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.757823 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758061 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.757871 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.758061 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.757994 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758073 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758106 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758126 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758225 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64flk\" (UniqueName: \"kubernetes.io/projected/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-kube-api-access-64flk\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.758289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758256 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758287 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkbfj\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-kube-api-access-xkbfj\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758350 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.758524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758503 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.758584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.758559 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.759086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.759069 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.760180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.760150 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.782887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.782849 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64flk\" (UniqueName: \"kubernetes.io/projected/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-kube-api-access-64flk\") pod \"maas-api-revoke-sub-76549406-jcnmz\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.860115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860030 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860078 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkbfj\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-kube-api-access-xkbfj\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860114 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860229 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860292 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860364 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860390 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860484 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860516 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860618 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-data\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860649 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-credential-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860732 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-socket\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.860887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.860866 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-certs\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.861082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.861050 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istiod-ca-cert\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.862699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.862678 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-podinfo\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.862752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.862704 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-envoy\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.881816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.881791 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:46.887712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.887694 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkbfj\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-kube-api-access-xkbfj\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.887805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.887744 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-token\") pod \"e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:46.919537 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:46.919480 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:47.191650 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:47.191620 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6690e865_0033_46a8_9c5c_93c1c3cb9ce5.slice/crio-410794c7e8703d64a3c06007b5abda3c61c7398dd4fb83877567c9e0432a193a WatchSource:0}: Error finding container 410794c7e8703d64a3c06007b5abda3c61c7398dd4fb83877567c9e0432a193a: Status 404 returned error can't find the container with id 410794c7e8703d64a3c06007b5abda3c61c7398dd4fb83877567c9e0432a193a Oct 31 21:45:47.192940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.192910 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz"] Oct 31 21:45:47.251477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.251452 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl"] Oct 31 21:45:47.252225 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:47.252193 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod362cb142_fbaf_4000_8fbc_be3f42e9a3cb.slice/crio-2b9f24102144c6de82d37efd70ac95e8f26b561b8b7d23613d0804b958ab46f2 WatchSource:0}: Error finding container 2b9f24102144c6de82d37efd70ac95e8f26b561b8b7d23613d0804b958ab46f2: Status 404 returned error can't find the container with id 2b9f24102144c6de82d37efd70ac95e8f26b561b8b7d23613d0804b958ab46f2 Oct 31 21:45:47.254835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.254773 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:45:47.254835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.254829 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:45:47.254975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.254857 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:45:47.837843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.837756 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp"] Oct 31 21:45:47.841995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.841971 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:47.860650 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.860615 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp"] Oct 31 21:45:47.971513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.971477 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" event={"ID":"6690e865-0033-46a8-9c5c-93c1c3cb9ce5","Type":"ContainerStarted","Data":"31d93040c8ba8d74456d8050440e518d23b40dd24ec5aa9fa5aa6feb0b2a35cf"} Oct 31 21:45:47.971691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.971540 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" event={"ID":"6690e865-0033-46a8-9c5c-93c1c3cb9ce5","Type":"ContainerStarted","Data":"410794c7e8703d64a3c06007b5abda3c61c7398dd4fb83877567c9e0432a193a"} Oct 31 21:45:47.973759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.973720 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" event={"ID":"362cb142-fbaf-4000-8fbc-be3f42e9a3cb","Type":"ContainerStarted","Data":"d55bd8882eabc327dba490dc10ddec3db0032dc678f14dbfa7201fc6803fb508"} Oct 31 21:45:47.973911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.973777 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" event={"ID":"362cb142-fbaf-4000-8fbc-be3f42e9a3cb","Type":"ContainerStarted","Data":"2b9f24102144c6de82d37efd70ac95e8f26b561b8b7d23613d0804b958ab46f2"} Oct 31 21:45:47.974212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.974142 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14209b83-acb4-4b94-8166-8b295654577f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:47.974431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.974407 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14209b83-acb4-4b94-8166-8b295654577f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:47.974495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:47.974473 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmfxg\" (UniqueName: \"kubernetes.io/projected/14209b83-acb4-4b94-8166-8b295654577f-kube-api-access-zmfxg\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.011378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.011325 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" podStartSLOduration=2.01129557 podStartE2EDuration="2.01129557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:48.011010752 +0000 UTC m=+1788.136117983" watchObservedRunningTime="2026-10-31 21:45:48.01129557 +0000 UTC m=+1788.136402802" Oct 31 21:45:48.059749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.059703 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podStartSLOduration=2.059691299 podStartE2EDuration="2.059691299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:48.058567556 +0000 UTC m=+1788.183674785" watchObservedRunningTime="2026-10-31 21:45:48.059691299 +0000 UTC m=+1788.184798529" Oct 31 21:45:48.075540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.075506 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14209b83-acb4-4b94-8166-8b295654577f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.075701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.075562 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmfxg\" (UniqueName: \"kubernetes.io/projected/14209b83-acb4-4b94-8166-8b295654577f-kube-api-access-zmfxg\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.075766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.075733 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14209b83-acb4-4b94-8166-8b295654577f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.076350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.076295 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14209b83-acb4-4b94-8166-8b295654577f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.078064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.078046 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14209b83-acb4-4b94-8166-8b295654577f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.099576 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.099494 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmfxg\" (UniqueName: \"kubernetes.io/projected/14209b83-acb4-4b94-8166-8b295654577f-kube-api-access-zmfxg\") pod \"maas-api-revoke-sub-b6b20bd4-mczqp\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.156055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.156023 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:48.500447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.500405 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp"] Oct 31 21:45:48.505089 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:45:48.505060 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14209b83_acb4_4b94_8166_8b295654577f.slice/crio-a5e5b32ac3d229311cad1f7056a5ca82bdc5b3502d55df2006d6d23bf6f1b69d WatchSource:0}: Error finding container a5e5b32ac3d229311cad1f7056a5ca82bdc5b3502d55df2006d6d23bf6f1b69d: Status 404 returned error can't find the container with id a5e5b32ac3d229311cad1f7056a5ca82bdc5b3502d55df2006d6d23bf6f1b69d Oct 31 21:45:48.920130 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.920104 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:48.921733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.921711 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 21:45:48.921840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:48.921756 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 21:45:49.005478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:49.005446 4294 generic.go:356] "Generic (PLEG): container finished" podID="6690e865-0033-46a8-9c5c-93c1c3cb9ce5" containerID="31d93040c8ba8d74456d8050440e518d23b40dd24ec5aa9fa5aa6feb0b2a35cf" exitCode=0 Oct 31 21:45:49.005661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:49.005525 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" event={"ID":"6690e865-0033-46a8-9c5c-93c1c3cb9ce5","Type":"ContainerDied","Data":"31d93040c8ba8d74456d8050440e518d23b40dd24ec5aa9fa5aa6feb0b2a35cf"} Oct 31 21:45:49.007094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:49.007069 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" event={"ID":"14209b83-acb4-4b94-8166-8b295654577f","Type":"ContainerStarted","Data":"71faf1923788afd02fba3ac5cfcd1f0c7685fd510d744d18d6c8d1f6fcb80b9a"} Oct 31 21:45:49.007207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:49.007111 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" event={"ID":"14209b83-acb4-4b94-8166-8b295654577f","Type":"ContainerStarted","Data":"a5e5b32ac3d229311cad1f7056a5ca82bdc5b3502d55df2006d6d23bf6f1b69d"} Oct 31 21:45:49.106616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:49.106499 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" podStartSLOduration=2.106474006 podStartE2EDuration="2.106474006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:45:49.098412499 +0000 UTC m=+1789.223519732" watchObservedRunningTime="2026-10-31 21:45:49.106474006 +0000 UTC m=+1789.231581238" Oct 31 21:45:49.920409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:49.920327 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 21:45:49.920409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:49.920378 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 21:45:50.015097 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.015067 4294 generic.go:356] "Generic (PLEG): container finished" podID="14209b83-acb4-4b94-8166-8b295654577f" containerID="71faf1923788afd02fba3ac5cfcd1f0c7685fd510d744d18d6c8d1f6fcb80b9a" exitCode=0 Oct 31 21:45:50.015280 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.015172 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" event={"ID":"14209b83-acb4-4b94-8166-8b295654577f","Type":"ContainerDied","Data":"71faf1923788afd02fba3ac5cfcd1f0c7685fd510d744d18d6c8d1f6fcb80b9a"} Oct 31 21:45:50.227131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.227102 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:50.227610 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.227590 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-openshift-ca\") pod \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " Oct 31 21:45:50.227728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.227716 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-kube-api-access-64flk\" (UniqueName: \"kubernetes.io/projected/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-kube-api-access-64flk\") pod \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " Oct 31 21:45:50.227930 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.227911 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-openshift-ca" pod "6690e865-0033-46a8-9c5c-93c1c3cb9ce5" (UID: "6690e865-0033-46a8-9c5c-93c1c3cb9ce5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:50.228185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.228166 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:50.229668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.229642 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-kube-api-access-64flk" pod "6690e865-0033-46a8-9c5c-93c1c3cb9ce5" (UID: "6690e865-0033-46a8-9c5c-93c1c3cb9ce5"). InnerVolumeSpecName "kube-api-access-64flk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:50.329623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.329584 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-xks-ca\") pod \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\" (UID: \"6690e865-0033-46a8-9c5c-93c1c3cb9ce5\") " Oct 31 21:45:50.330073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.330049 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64flk\" (UniqueName: \"kubernetes.io/projected/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-kube-api-access-64flk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:50.332333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.332034 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-xks-ca" pod "6690e865-0033-46a8-9c5c-93c1c3cb9ce5" (UID: "6690e865-0033-46a8-9c5c-93c1c3cb9ce5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:50.432044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.432004 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6690e865-0033-46a8-9c5c-93c1c3cb9ce5-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:50.920715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.920677 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 21:45:50.921089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:50.920728 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 21:45:51.023843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.023815 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" Oct 31 21:45:51.024414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.024390 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz" event={"ID":"6690e865-0033-46a8-9c5c-93c1c3cb9ce5","Type":"ContainerDied","Data":"410794c7e8703d64a3c06007b5abda3c61c7398dd4fb83877567c9e0432a193a"} Oct 31 21:45:51.024918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.024905 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410794c7e8703d64a3c06007b5abda3c61c7398dd4fb83877567c9e0432a193a" Oct 31 21:45:51.230778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.230756 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:51.242511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.242483 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14209b83-acb4-4b94-8166-8b295654577f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14209b83-acb4-4b94-8166-8b295654577f-maas-api-xks-ca\") pod \"14209b83-acb4-4b94-8166-8b295654577f\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " Oct 31 21:45:51.242635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.242547 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14209b83-acb4-4b94-8166-8b295654577f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14209b83-acb4-4b94-8166-8b295654577f-maas-api-openshift-ca\") pod \"14209b83-acb4-4b94-8166-8b295654577f\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " Oct 31 21:45:51.242701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.242662 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14209b83-acb4-4b94-8166-8b295654577f-kube-api-access-zmfxg\" (UniqueName: \"kubernetes.io/projected/14209b83-acb4-4b94-8166-8b295654577f-kube-api-access-zmfxg\") pod \"14209b83-acb4-4b94-8166-8b295654577f\" (UID: \"14209b83-acb4-4b94-8166-8b295654577f\") " Oct 31 21:45:51.242942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.242912 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14209b83-acb4-4b94-8166-8b295654577f-maas-api-openshift-ca" pod "14209b83-acb4-4b94-8166-8b295654577f" (UID: "14209b83-acb4-4b94-8166-8b295654577f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:45:51.245483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.244901 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14209b83-acb4-4b94-8166-8b295654577f-maas-api-xks-ca" pod "14209b83-acb4-4b94-8166-8b295654577f" (UID: "14209b83-acb4-4b94-8166-8b295654577f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:45:51.245483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.244945 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14209b83-acb4-4b94-8166-8b295654577f-kube-api-access-zmfxg" pod "14209b83-acb4-4b94-8166-8b295654577f" (UID: "14209b83-acb4-4b94-8166-8b295654577f"). InnerVolumeSpecName "kube-api-access-zmfxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:45:51.343803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.343756 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmfxg\" (UniqueName: \"kubernetes.io/projected/14209b83-acb4-4b94-8166-8b295654577f-kube-api-access-zmfxg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:51.343803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.343799 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14209b83-acb4-4b94-8166-8b295654577f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:51.343803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.343813 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14209b83-acb4-4b94-8166-8b295654577f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:45:51.920399 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.920368 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 21:45:51.920589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:51.920420 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 21:45:52.031345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:52.031316 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" event={"ID":"14209b83-acb4-4b94-8166-8b295654577f","Type":"ContainerDied","Data":"a5e5b32ac3d229311cad1f7056a5ca82bdc5b3502d55df2006d6d23bf6f1b69d"} Oct 31 21:45:52.031667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:52.031358 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp" Oct 31 21:45:52.031667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:52.031363 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5e5b32ac3d229311cad1f7056a5ca82bdc5b3502d55df2006d6d23bf6f1b69d" Oct 31 21:45:52.919971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:52.919920 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 21:45:52.920148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:52.919989 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 21:45:53.920096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:53.920052 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 31 21:45:53.920611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:53.920122 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 31 21:45:54.924607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:54.924573 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:54.924984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:54.924870 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:54.925705 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:54.925686 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:45:57.870326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:57.868427 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 21:45:57.975902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:57.970761 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:45:59.533031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.533000 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht"] Oct 31 21:45:59.533556 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.533541 4294 cpu_manager.go:414] "containerMap: removing container" podUID="14209b83-acb4-4b94-8166-8b295654577f" containerName="revoke-keys" Oct 31 21:45:59.533599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.533556 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14209b83-acb4-4b94-8166-8b295654577f" containerName="revoke-keys" Oct 31 21:45:59.533599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.533576 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6690e865-0033-46a8-9c5c-93c1c3cb9ce5" containerName="revoke-keys" Oct 31 21:45:59.533599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.533581 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6690e865-0033-46a8-9c5c-93c1c3cb9ce5" containerName="revoke-keys" Oct 31 21:45:59.533688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.533675 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6690e865-0033-46a8-9c5c-93c1c3cb9ce5" containerName="revoke-keys" Oct 31 21:45:59.533721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.533690 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="14209b83-acb4-4b94-8166-8b295654577f" containerName="revoke-keys" Oct 31 21:45:59.537073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.537057 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.556797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.556771 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht"] Oct 31 21:45:59.563404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.563379 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:45:59.563507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.563456 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:45:59.621401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.621364 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.621579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.621446 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.621579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.621526 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28ptn\" (UniqueName: \"kubernetes.io/projected/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-kube-api-access-28ptn\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.722837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.722800 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.722837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.722838 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28ptn\" (UniqueName: \"kubernetes.io/projected/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-kube-api-access-28ptn\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.723047 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.723028 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.723601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.723582 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.725313 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.725285 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.747859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.747831 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28ptn\" (UniqueName: \"kubernetes.io/projected/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-kube-api-access-28ptn\") pod \"maas-api-revoke-sub-475a1f35-h75ht\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:45:59.850748 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:45:59.850673 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:46:00.206901 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:00.200735 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4dbd5c_bad9_4834_ad4e_d84f20e3651f.slice/crio-9b3cb5fd05494b4c5ff95beb49bb53cba309a814563583631b6793bdfd78c997 WatchSource:0}: Error finding container 9b3cb5fd05494b4c5ff95beb49bb53cba309a814563583631b6793bdfd78c997: Status 404 returned error can't find the container with id 9b3cb5fd05494b4c5ff95beb49bb53cba309a814563583631b6793bdfd78c997 Oct 31 21:46:00.206901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.201826 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht"] Oct 31 21:46:00.229418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.229386 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" event={"ID":"de4dbd5c-bad9-4834-ad4e-d84f20e3651f","Type":"ContainerStarted","Data":"9b3cb5fd05494b4c5ff95beb49bb53cba309a814563583631b6793bdfd78c997"} Oct 31 21:46:00.233073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.233053 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:46:00.240366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.240347 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:46:00.834337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.834250 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw"] Oct 31 21:46:00.839782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.839765 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:00.858439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.858412 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw"] Oct 31 21:46:00.937710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.937679 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:00.937924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.937910 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5z6\" (UniqueName: \"kubernetes.io/projected/4532111c-91ee-4579-99a0-e7d2c877ccc9-kube-api-access-wj5z6\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:00.938165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:00.938147 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.038868 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.038834 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.039142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.039104 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.039245 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.039183 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5z6\" (UniqueName: \"kubernetes.io/projected/4532111c-91ee-4579-99a0-e7d2c877ccc9-kube-api-access-wj5z6\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.040184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.040163 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.041401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.041385 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.062968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.062946 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5z6\" (UniqueName: \"kubernetes.io/projected/4532111c-91ee-4579-99a0-e7d2c877ccc9-kube-api-access-wj5z6\") pod \"maas-api-revoke-sub-cc9ca4ce-z7wdw\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.154270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.154185 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:01.239409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.239367 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" event={"ID":"de4dbd5c-bad9-4834-ad4e-d84f20e3651f","Type":"ContainerStarted","Data":"f408e23cd8041c422d11e97b30aea120162fa260195ddc56791cc104e9bc0df3"} Oct 31 21:46:01.334083 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:46:01.334050 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:46:01.496327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.494796 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" podStartSLOduration=2.494774644 podStartE2EDuration="2.494774644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:01.298682837 +0000 UTC m=+1801.423790070" watchObservedRunningTime="2026-10-31 21:46:01.494774644 +0000 UTC m=+1801.619881878" Oct 31 21:46:01.499542 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:01.499508 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw"] Oct 31 21:46:01.499747 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:01.499725 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4532111c_91ee_4579_99a0_e7d2c877ccc9.slice/crio-e8247371e49228081c7d547449ce9b36e3b9c8499bc42722a22ae24b5cdc74c7 WatchSource:0}: Error finding container e8247371e49228081c7d547449ce9b36e3b9c8499bc42722a22ae24b5cdc74c7: Status 404 returned error can't find the container with id e8247371e49228081c7d547449ce9b36e3b9c8499bc42722a22ae24b5cdc74c7 Oct 31 21:46:02.247038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:02.247001 4294 generic.go:356] "Generic (PLEG): container finished" podID="de4dbd5c-bad9-4834-ad4e-d84f20e3651f" containerID="f408e23cd8041c422d11e97b30aea120162fa260195ddc56791cc104e9bc0df3" exitCode=0 Oct 31 21:46:02.247500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:02.247087 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" event={"ID":"de4dbd5c-bad9-4834-ad4e-d84f20e3651f","Type":"ContainerDied","Data":"f408e23cd8041c422d11e97b30aea120162fa260195ddc56791cc104e9bc0df3"} Oct 31 21:46:02.249165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:02.249131 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" event={"ID":"4532111c-91ee-4579-99a0-e7d2c877ccc9","Type":"ContainerStarted","Data":"5f6db778fef008a38791068293370ee9454499beab7175762e80d435ee090aae"} Oct 31 21:46:02.249269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:02.249189 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" event={"ID":"4532111c-91ee-4579-99a0-e7d2c877ccc9","Type":"ContainerStarted","Data":"e8247371e49228081c7d547449ce9b36e3b9c8499bc42722a22ae24b5cdc74c7"} Oct 31 21:46:02.331355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:02.331278 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" podStartSLOduration=2.331263849 podStartE2EDuration="2.331263849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:02.329425738 +0000 UTC m=+1802.454532971" watchObservedRunningTime="2026-10-31 21:46:02.331263849 +0000 UTC m=+1802.456371079" Oct 31 21:46:03.259440 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.259406 4294 generic.go:356] "Generic (PLEG): container finished" podID="4532111c-91ee-4579-99a0-e7d2c877ccc9" containerID="5f6db778fef008a38791068293370ee9454499beab7175762e80d435ee090aae" exitCode=0 Oct 31 21:46:03.259934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.259494 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" event={"ID":"4532111c-91ee-4579-99a0-e7d2c877ccc9","Type":"ContainerDied","Data":"5f6db778fef008a38791068293370ee9454499beab7175762e80d435ee090aae"} Oct 31 21:46:03.457605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.457577 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:46:03.622247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.622157 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-xks-ca\") pod \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " Oct 31 21:46:03.622450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.622356 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-kube-api-access-28ptn\" (UniqueName: \"kubernetes.io/projected/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-kube-api-access-28ptn\") pod \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " Oct 31 21:46:03.622450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.622417 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-openshift-ca\") pod \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\" (UID: \"de4dbd5c-bad9-4834-ad4e-d84f20e3651f\") " Oct 31 21:46:03.622825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.622793 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-openshift-ca" pod "de4dbd5c-bad9-4834-ad4e-d84f20e3651f" (UID: "de4dbd5c-bad9-4834-ad4e-d84f20e3651f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:03.623150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.623130 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:03.624696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.624663 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-xks-ca" pod "de4dbd5c-bad9-4834-ad4e-d84f20e3651f" (UID: "de4dbd5c-bad9-4834-ad4e-d84f20e3651f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:03.624696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.624667 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-kube-api-access-28ptn" pod "de4dbd5c-bad9-4834-ad4e-d84f20e3651f" (UID: "de4dbd5c-bad9-4834-ad4e-d84f20e3651f"). InnerVolumeSpecName "kube-api-access-28ptn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:03.724163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.724122 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28ptn\" (UniqueName: \"kubernetes.io/projected/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-kube-api-access-28ptn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:03.724163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:03.724162 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de4dbd5c-bad9-4834-ad4e-d84f20e3651f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:04.267029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.266932 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" event={"ID":"de4dbd5c-bad9-4834-ad4e-d84f20e3651f","Type":"ContainerDied","Data":"9b3cb5fd05494b4c5ff95beb49bb53cba309a814563583631b6793bdfd78c997"} Oct 31 21:46:04.267029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.266992 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b3cb5fd05494b4c5ff95beb49bb53cba309a814563583631b6793bdfd78c997" Oct 31 21:46:04.267029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.266954 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht" Oct 31 21:46:04.407181 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.407153 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:04.540156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.540080 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-openshift-ca\") pod \"4532111c-91ee-4579-99a0-e7d2c877ccc9\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " Oct 31 21:46:04.540343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.540236 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4532111c-91ee-4579-99a0-e7d2c877ccc9-kube-api-access-wj5z6\" (UniqueName: \"kubernetes.io/projected/4532111c-91ee-4579-99a0-e7d2c877ccc9-kube-api-access-wj5z6\") pod \"4532111c-91ee-4579-99a0-e7d2c877ccc9\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " Oct 31 21:46:04.540343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.540276 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-xks-ca\") pod \"4532111c-91ee-4579-99a0-e7d2c877ccc9\" (UID: \"4532111c-91ee-4579-99a0-e7d2c877ccc9\") " Oct 31 21:46:04.540499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.540475 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-openshift-ca" pod "4532111c-91ee-4579-99a0-e7d2c877ccc9" (UID: "4532111c-91ee-4579-99a0-e7d2c877ccc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:04.540937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.540911 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:04.542445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.542419 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4532111c-91ee-4579-99a0-e7d2c877ccc9-kube-api-access-wj5z6" pod "4532111c-91ee-4579-99a0-e7d2c877ccc9" (UID: "4532111c-91ee-4579-99a0-e7d2c877ccc9"). InnerVolumeSpecName "kube-api-access-wj5z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:04.542445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.542418 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-xks-ca" pod "4532111c-91ee-4579-99a0-e7d2c877ccc9" (UID: "4532111c-91ee-4579-99a0-e7d2c877ccc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:04.642066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.642025 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5z6\" (UniqueName: \"kubernetes.io/projected/4532111c-91ee-4579-99a0-e7d2c877ccc9-kube-api-access-wj5z6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:04.642066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:04.642057 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4532111c-91ee-4579-99a0-e7d2c877ccc9-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:05.275802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:05.275771 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" Oct 31 21:46:05.276281 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:05.275807 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw" event={"ID":"4532111c-91ee-4579-99a0-e7d2c877ccc9","Type":"ContainerDied","Data":"e8247371e49228081c7d547449ce9b36e3b9c8499bc42722a22ae24b5cdc74c7"} Oct 31 21:46:05.276281 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:05.275856 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8247371e49228081c7d547449ce9b36e3b9c8499bc42722a22ae24b5cdc74c7" Oct 31 21:46:06.283552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.283521 4294 generic.go:356] "Generic (PLEG): container finished" podID="134fced6-b695-470e-b923-d59215873872" containerID="52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb" exitCode=0 Oct 31 21:46:06.284031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.283599 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" event={"ID":"134fced6-b695-470e-b923-d59215873872","Type":"ContainerDied","Data":"52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb"} Oct 31 21:46:06.284031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.283995 4294 scope.go:122] "RemoveContainer" containerID="52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb" Oct 31 21:46:06.285478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.285454 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:06.285541 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.285519 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:06.285589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.285578 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:06.369821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.369794 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27"] Oct 31 21:46:06.370680 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.370656 4294 cpu_manager.go:414] "containerMap: removing container" podUID="de4dbd5c-bad9-4834-ad4e-d84f20e3651f" containerName="revoke-keys" Oct 31 21:46:06.370796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.370683 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4dbd5c-bad9-4834-ad4e-d84f20e3651f" containerName="revoke-keys" Oct 31 21:46:06.370796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.370717 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4532111c-91ee-4579-99a0-e7d2c877ccc9" containerName="revoke-keys" Oct 31 21:46:06.370796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.370727 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4532111c-91ee-4579-99a0-e7d2c877ccc9" containerName="revoke-keys" Oct 31 21:46:06.370960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.370863 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4dbd5c-bad9-4834-ad4e-d84f20e3651f" containerName="revoke-keys" Oct 31 21:46:06.370960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.370884 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4532111c-91ee-4579-99a0-e7d2c877ccc9" containerName="revoke-keys" Oct 31 21:46:06.374935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.374913 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.389758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.389733 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:46:06.389876 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.389842 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:46:06.391609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.391574 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27"] Oct 31 21:46:06.570822 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.570741 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.570822 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.570791 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62fc\" (UniqueName: \"kubernetes.io/projected/881bbd23-8801-42cc-8d8c-1f57746d952f-kube-api-access-n62fc\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.571027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.570971 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.672877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.672838 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.673067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.673049 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.673122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.673107 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62fc\" (UniqueName: \"kubernetes.io/projected/881bbd23-8801-42cc-8d8c-1f57746d952f-kube-api-access-n62fc\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.673694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.673669 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.675186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.675159 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.696284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.696262 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62fc\" (UniqueName: \"kubernetes.io/projected/881bbd23-8801-42cc-8d8c-1f57746d952f-kube-api-access-n62fc\") pod \"maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:06.704160 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:06.704134 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:07.019894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:07.019480 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27"] Oct 31 21:46:07.303289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:07.303202 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" event={"ID":"134fced6-b695-470e-b923-d59215873872","Type":"ContainerStarted","Data":"2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939"} Oct 31 21:46:07.304634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:07.304609 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" event={"ID":"881bbd23-8801-42cc-8d8c-1f57746d952f","Type":"ContainerStarted","Data":"407448e97fed162eb29902558fa96e1433d2b505514f5aac61495aaa93c855ec"} Oct 31 21:46:07.905554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:07.905522 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:46:07.905889 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:07.905867 4294 patch_prober.go:29] interesting pod/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:46:07.905935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:07.905916 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 21:46:08.316282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:08.316252 4294 generic.go:356] "Generic (PLEG): container finished" podID="881bbd23-8801-42cc-8d8c-1f57746d952f" containerID="f0def5150250ad8b6e064a8e17dce6eada1afa9b8b7599dbe5ff87213a22a97a" exitCode=0 Oct 31 21:46:08.316772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:08.316362 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" event={"ID":"881bbd23-8801-42cc-8d8c-1f57746d952f","Type":"ContainerDied","Data":"f0def5150250ad8b6e064a8e17dce6eada1afa9b8b7599dbe5ff87213a22a97a"} Oct 31 21:46:08.906431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:08.906392 4294 patch_prober.go:29] interesting pod/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:46:08.906619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:08.906444 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 21:46:09.531158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.531132 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:09.625499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.625418 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-xks-ca\") pod \"881bbd23-8801-42cc-8d8c-1f57746d952f\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " Oct 31 21:46:09.625665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.625516 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-openshift-ca\") pod \"881bbd23-8801-42cc-8d8c-1f57746d952f\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " Oct 31 21:46:09.625665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.625557 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881bbd23-8801-42cc-8d8c-1f57746d952f-kube-api-access-n62fc\" (UniqueName: \"kubernetes.io/projected/881bbd23-8801-42cc-8d8c-1f57746d952f-kube-api-access-n62fc\") pod \"881bbd23-8801-42cc-8d8c-1f57746d952f\" (UID: \"881bbd23-8801-42cc-8d8c-1f57746d952f\") " Oct 31 21:46:09.625930 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.625902 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-openshift-ca" pod "881bbd23-8801-42cc-8d8c-1f57746d952f" (UID: "881bbd23-8801-42cc-8d8c-1f57746d952f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:09.626336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.626313 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:09.628949 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.628918 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881bbd23-8801-42cc-8d8c-1f57746d952f-kube-api-access-n62fc" pod "881bbd23-8801-42cc-8d8c-1f57746d952f" (UID: "881bbd23-8801-42cc-8d8c-1f57746d952f"). InnerVolumeSpecName "kube-api-access-n62fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:09.638315 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.631060 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-xks-ca" pod "881bbd23-8801-42cc-8d8c-1f57746d952f" (UID: "881bbd23-8801-42cc-8d8c-1f57746d952f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:09.735334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.732739 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62fc\" (UniqueName: \"kubernetes.io/projected/881bbd23-8801-42cc-8d8c-1f57746d952f-kube-api-access-n62fc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:09.735334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.732775 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/881bbd23-8801-42cc-8d8c-1f57746d952f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:09.905727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.905643 4294 patch_prober.go:29] interesting pod/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:46:09.905727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:09.905698 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 21:46:10.353128 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:10.353085 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" event={"ID":"881bbd23-8801-42cc-8d8c-1f57746d952f","Type":"ContainerDied","Data":"407448e97fed162eb29902558fa96e1433d2b505514f5aac61495aaa93c855ec"} Oct 31 21:46:10.357446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:10.357423 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="407448e97fed162eb29902558fa96e1433d2b505514f5aac61495aaa93c855ec" Oct 31 21:46:10.362677 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:10.362653 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27" Oct 31 21:46:10.914326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:10.910689 4294 patch_prober.go:29] interesting pod/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:46:10.914326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:10.910773 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 21:46:11.908792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:11.908756 4294 patch_prober.go:29] interesting pod/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 21:46:11.909151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:11.909103 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 21:46:12.947887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:12.947847 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:46:12.948387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:12.948359 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:46:12.949895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:12.949872 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:46:13.025854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.025822 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp"] Oct 31 21:46:13.028831 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.028807 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-09f39e-ndzcp"] Oct 31 21:46:13.267530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.267491 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp"] Oct 31 21:46:13.268093 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.268074 4294 cpu_manager.go:414] "containerMap: removing container" podUID="881bbd23-8801-42cc-8d8c-1f57746d952f" containerName="revoke-keys" Oct 31 21:46:13.268202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.268095 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881bbd23-8801-42cc-8d8c-1f57746d952f" containerName="revoke-keys" Oct 31 21:46:13.268280 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.268266 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="881bbd23-8801-42cc-8d8c-1f57746d952f" containerName="revoke-keys" Oct 31 21:46:13.271835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.271814 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.289343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.288393 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:46:13.289343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.288594 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:46:13.292385 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.292366 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp"] Oct 31 21:46:13.348635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.348605 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.348886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.348865 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsqg\" (UniqueName: \"kubernetes.io/projected/ea1b1051-c8bd-4829-8b59-a999a9807a0b-kube-api-access-cbsqg\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.348972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.348962 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.450376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.450341 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsqg\" (UniqueName: \"kubernetes.io/projected/ea1b1051-c8bd-4829-8b59-a999a9807a0b-kube-api-access-cbsqg\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.450559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.450425 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.450559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.450487 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.451151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.451128 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.453082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.453059 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.473846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.473823 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsqg\" (UniqueName: \"kubernetes.io/projected/ea1b1051-c8bd-4829-8b59-a999a9807a0b-kube-api-access-cbsqg\") pod \"maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.585566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.585493 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:13.891477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.891445 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp"] Oct 31 21:46:13.893003 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:13.892974 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1b1051_c8bd_4829_8b59_a999a9807a0b.slice/crio-451afb4872c5d3a0d774363b94f4967b96a356b466cab02dc8aad5c58478ed4d WatchSource:0}: Error finding container 451afb4872c5d3a0d774363b94f4967b96a356b466cab02dc8aad5c58478ed4d: Status 404 returned error can't find the container with id 451afb4872c5d3a0d774363b94f4967b96a356b466cab02dc8aad5c58478ed4d Oct 31 21:46:13.989769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:13.989736 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c776ea10-f633-4145-8801-970826053d31" path="/var/lib/kubelet/pods/c776ea10-f633-4145-8801-970826053d31/volumes" Oct 31 21:46:14.488680 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:14.488643 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" event={"ID":"ea1b1051-c8bd-4829-8b59-a999a9807a0b","Type":"ContainerStarted","Data":"b4a532e8565aa03496d6dbd8873c3db4fc9421ddb50d36b650e26ffe16752a14"} Oct 31 21:46:14.488869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:14.488696 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" event={"ID":"ea1b1051-c8bd-4829-8b59-a999a9807a0b","Type":"ContainerStarted","Data":"451afb4872c5d3a0d774363b94f4967b96a356b466cab02dc8aad5c58478ed4d"} Oct 31 21:46:14.529091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:14.529047 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" podStartSLOduration=1.529033164 podStartE2EDuration="1.529033164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:14.528196241 +0000 UTC m=+1814.653303472" watchObservedRunningTime="2026-10-31 21:46:14.529033164 +0000 UTC m=+1814.654140419" Oct 31 21:46:15.497167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:15.497129 4294 generic.go:356] "Generic (PLEG): container finished" podID="ea1b1051-c8bd-4829-8b59-a999a9807a0b" containerID="b4a532e8565aa03496d6dbd8873c3db4fc9421ddb50d36b650e26ffe16752a14" exitCode=0 Oct 31 21:46:15.497575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:15.497207 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" event={"ID":"ea1b1051-c8bd-4829-8b59-a999a9807a0b","Type":"ContainerDied","Data":"b4a532e8565aa03496d6dbd8873c3db4fc9421ddb50d36b650e26ffe16752a14"} Oct 31 21:46:16.664909 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.664883 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:16.723657 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.723624 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw"] Oct 31 21:46:16.724186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.724165 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ea1b1051-c8bd-4829-8b59-a999a9807a0b" containerName="revoke-keys" Oct 31 21:46:16.724186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.724183 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1b1051-c8bd-4829-8b59-a999a9807a0b" containerName="revoke-keys" Oct 31 21:46:16.724384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.724365 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1b1051-c8bd-4829-8b59-a999a9807a0b" containerName="revoke-keys" Oct 31 21:46:16.728123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.728105 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.748580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.748551 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw"] Oct 31 21:46:16.791558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.791462 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-xks-ca\") pod \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " Oct 31 21:46:16.793977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.793104 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-openshift-ca\") pod \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " Oct 31 21:46:16.793977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.793187 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1b1051-c8bd-4829-8b59-a999a9807a0b-kube-api-access-cbsqg\" (UniqueName: \"kubernetes.io/projected/ea1b1051-c8bd-4829-8b59-a999a9807a0b-kube-api-access-cbsqg\") pod \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\" (UID: \"ea1b1051-c8bd-4829-8b59-a999a9807a0b\") " Oct 31 21:46:16.794206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.794182 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-openshift-ca" pod "ea1b1051-c8bd-4829-8b59-a999a9807a0b" (UID: "ea1b1051-c8bd-4829-8b59-a999a9807a0b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:16.795755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.795727 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-xks-ca" pod "ea1b1051-c8bd-4829-8b59-a999a9807a0b" (UID: "ea1b1051-c8bd-4829-8b59-a999a9807a0b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:16.796707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.796669 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1b1051-c8bd-4829-8b59-a999a9807a0b-kube-api-access-cbsqg" pod "ea1b1051-c8bd-4829-8b59-a999a9807a0b" (UID: "ea1b1051-c8bd-4829-8b59-a999a9807a0b"). InnerVolumeSpecName "kube-api-access-cbsqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:16.895284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.895252 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65z59\" (UniqueName: \"kubernetes.io/projected/ece0dd47-f5af-4e69-ba8a-4c1985074337-kube-api-access-65z59\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.895479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.895437 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.895567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.895549 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.895727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.895709 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:16.895770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.895731 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1b1051-c8bd-4829-8b59-a999a9807a0b-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:16.895770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.895742 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbsqg\" (UniqueName: \"kubernetes.io/projected/ea1b1051-c8bd-4829-8b59-a999a9807a0b-kube-api-access-cbsqg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:16.997263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.997227 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.997478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.997355 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65z59\" (UniqueName: \"kubernetes.io/projected/ece0dd47-f5af-4e69-ba8a-4c1985074337-kube-api-access-65z59\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.997478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.997441 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.997946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.997920 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:16.999776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:16.999747 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:17.020855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:17.020831 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65z59\" (UniqueName: \"kubernetes.io/projected/ece0dd47-f5af-4e69-ba8a-4c1985074337-kube-api-access-65z59\") pod \"maas-api-revoke-sub-01c6bcee-nrlvw\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:17.052777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:17.052724 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:17.411091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:17.411064 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw"] Oct 31 21:46:17.411425 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:17.411396 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece0dd47_f5af_4e69_ba8a_4c1985074337.slice/crio-6bddf06f2c69afdf5b02ffb9a7e1ffccebfae5bc58a8332a9dbafed7b0dad768 WatchSource:0}: Error finding container 6bddf06f2c69afdf5b02ffb9a7e1ffccebfae5bc58a8332a9dbafed7b0dad768: Status 404 returned error can't find the container with id 6bddf06f2c69afdf5b02ffb9a7e1ffccebfae5bc58a8332a9dbafed7b0dad768 Oct 31 21:46:17.513485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:17.513444 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" event={"ID":"ece0dd47-f5af-4e69-ba8a-4c1985074337","Type":"ContainerStarted","Data":"6bddf06f2c69afdf5b02ffb9a7e1ffccebfae5bc58a8332a9dbafed7b0dad768"} Oct 31 21:46:17.514933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:17.514908 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" Oct 31 21:46:17.515059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:17.514949 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp" event={"ID":"ea1b1051-c8bd-4829-8b59-a999a9807a0b","Type":"ContainerDied","Data":"451afb4872c5d3a0d774363b94f4967b96a356b466cab02dc8aad5c58478ed4d"} Oct 31 21:46:17.515059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:17.515005 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="451afb4872c5d3a0d774363b94f4967b96a356b466cab02dc8aad5c58478ed4d" Oct 31 21:46:18.523034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:18.523002 4294 generic.go:356] "Generic (PLEG): container finished" podID="ece0dd47-f5af-4e69-ba8a-4c1985074337" containerID="a37b26562fd7c5a2691f1c27275e56645ed0c52b8227dc6128c075e55ee06c00" exitCode=0 Oct 31 21:46:18.523435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:18.523094 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" event={"ID":"ece0dd47-f5af-4e69-ba8a-4c1985074337","Type":"ContainerDied","Data":"a37b26562fd7c5a2691f1c27275e56645ed0c52b8227dc6128c075e55ee06c00"} Oct 31 21:46:19.673467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.673440 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:19.836385 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.836272 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-openshift-ca\") pod \"ece0dd47-f5af-4e69-ba8a-4c1985074337\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " Oct 31 21:46:19.836582 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.836450 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece0dd47-f5af-4e69-ba8a-4c1985074337-kube-api-access-65z59\" (UniqueName: \"kubernetes.io/projected/ece0dd47-f5af-4e69-ba8a-4c1985074337-kube-api-access-65z59\") pod \"ece0dd47-f5af-4e69-ba8a-4c1985074337\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " Oct 31 21:46:19.836582 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.836531 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-xks-ca\") pod \"ece0dd47-f5af-4e69-ba8a-4c1985074337\" (UID: \"ece0dd47-f5af-4e69-ba8a-4c1985074337\") " Oct 31 21:46:19.836701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.836623 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-openshift-ca" pod "ece0dd47-f5af-4e69-ba8a-4c1985074337" (UID: "ece0dd47-f5af-4e69-ba8a-4c1985074337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:19.837157 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.837123 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:19.838664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.838630 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-xks-ca" pod "ece0dd47-f5af-4e69-ba8a-4c1985074337" (UID: "ece0dd47-f5af-4e69-ba8a-4c1985074337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:19.838789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.838740 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece0dd47-f5af-4e69-ba8a-4c1985074337-kube-api-access-65z59" pod "ece0dd47-f5af-4e69-ba8a-4c1985074337" (UID: "ece0dd47-f5af-4e69-ba8a-4c1985074337"). InnerVolumeSpecName "kube-api-access-65z59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:19.942345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.940283 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65z59\" (UniqueName: \"kubernetes.io/projected/ece0dd47-f5af-4e69-ba8a-4c1985074337-kube-api-access-65z59\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:19.942345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:19.940339 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ece0dd47-f5af-4e69-ba8a-4c1985074337-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:20.132158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:20.132070 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm"] Oct 31 21:46:20.132435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:20.132384 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" containerID="cri-o://8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911" gracePeriod=30 Oct 31 21:46:20.537524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:20.537492 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" Oct 31 21:46:20.537678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:20.537529 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw" event={"ID":"ece0dd47-f5af-4e69-ba8a-4c1985074337","Type":"ContainerDied","Data":"6bddf06f2c69afdf5b02ffb9a7e1ffccebfae5bc58a8332a9dbafed7b0dad768"} Oct 31 21:46:20.537678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:20.537583 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bddf06f2c69afdf5b02ffb9a7e1ffccebfae5bc58a8332a9dbafed7b0dad768" Oct 31 21:46:20.594368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:20.594337 4294 patch_prober.go:29] interesting pod/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:46:20.594530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:20.594382 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:46:21.830037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.829996 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k"] Oct 31 21:46:21.830797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.830776 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ece0dd47-f5af-4e69-ba8a-4c1985074337" containerName="revoke-keys" Oct 31 21:46:21.830908 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.830799 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece0dd47-f5af-4e69-ba8a-4c1985074337" containerName="revoke-keys" Oct 31 21:46:21.830964 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.830931 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece0dd47-f5af-4e69-ba8a-4c1985074337" containerName="revoke-keys" Oct 31 21:46:21.841595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.836750 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.856465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.856436 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k"] Oct 31 21:46:21.968907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.968870 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583ea135-1351-4fd2-a308-42bf4fab268e-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969018 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969052 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969070 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583ea135-1351-4fd2-a308-42bf4fab268e-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969176 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969254 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969276 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969375 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:21.969486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:21.969431 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbnvn\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-kube-api-access-zbnvn\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.070705 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.070669 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.070886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.070739 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.070886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.070761 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.070886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.070791 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.070886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.070825 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbnvn\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-kube-api-access-zbnvn\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.070924 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583ea135-1351-4fd2-a308-42bf4fab268e-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071226 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071226 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071433 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071468 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071492 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583ea135-1351-4fd2-a308-42bf4fab268e-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071637 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583ea135-1351-4fd2-a308-42bf4fab268e-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071797 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.071884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.071827 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.073228 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.073204 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.073956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.073938 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583ea135-1351-4fd2-a308-42bf4fab268e-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.094653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.094583 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.094808 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.094790 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbnvn\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-kube-api-access-zbnvn\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.123211 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.122612 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf"] Oct 31 21:46:22.128136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.128114 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.154359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.153102 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b01ff6-openshift-default-dockercfg-v5gpl\"" Oct 31 21:46:22.159457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.159427 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf"] Oct 31 21:46:22.159641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.159612 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:22.173351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173321 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-envoy\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173386 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-data\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173455 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-podinfo\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173499 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-credential-socket\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173569 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istiod-ca-cert\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173713 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-token\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173749 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-certs\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173780 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-socket\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.173832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.173803 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmvq\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-kube-api-access-dwmvq\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275296 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-envoy\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275379 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-data\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275424 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-podinfo\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275464 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-credential-socket\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275525 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istiod-ca-cert\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275627 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-token\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275661 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-certs\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275692 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-socket\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.276344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.275718 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmvq\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-kube-api-access-dwmvq\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.278282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.277696 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istiod-ca-cert\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.278282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.277940 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-data\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.279224 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.278726 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-socket\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.279224 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.278952 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-credential-socket\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.279664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.279645 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-certs\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.281142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.281114 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-podinfo\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.282079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.282039 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-envoy\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.298970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.298945 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmvq\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-kube-api-access-dwmvq\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.301914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.301892 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-token\") pod \"e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.454223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.453602 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:22.520258 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:22.520228 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583ea135_1351_4fd2_a308_42bf4fab268e.slice/crio-8cb53353e8c25404ee0ec15e703337eed8054e4773eed4ecae1b58a2a242e6c9 WatchSource:0}: Error finding container 8cb53353e8c25404ee0ec15e703337eed8054e4773eed4ecae1b58a2a242e6c9: Status 404 returned error can't find the container with id 8cb53353e8c25404ee0ec15e703337eed8054e4773eed4ecae1b58a2a242e6c9 Oct 31 21:46:22.520551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.520527 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k"] Oct 31 21:46:22.522967 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.522927 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:22.523067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.522994 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:22.523067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.523025 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:22.561751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.561714 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" event={"ID":"583ea135-1351-4fd2-a308-42bf4fab268e","Type":"ContainerStarted","Data":"8cb53353e8c25404ee0ec15e703337eed8054e4773eed4ecae1b58a2a242e6c9"} Oct 31 21:46:22.800268 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:22.800236 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cc3f4e_9300_479c_b129_c0f12eed77f1.slice/crio-c550af9362c07ad83830c617194ee7f717f264d6a82652464e94259da224cbcd WatchSource:0}: Error finding container c550af9362c07ad83830c617194ee7f717f264d6a82652464e94259da224cbcd: Status 404 returned error can't find the container with id c550af9362c07ad83830c617194ee7f717f264d6a82652464e94259da224cbcd Oct 31 21:46:22.800691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.800666 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf"] Oct 31 21:46:22.803619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.803585 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:22.803716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.803667 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:22.803772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:22.803723 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:23.570274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:23.570229 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" event={"ID":"f4cc3f4e-9300-479c-b129-c0f12eed77f1","Type":"ContainerStarted","Data":"854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22"} Oct 31 21:46:23.570747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:23.570315 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" event={"ID":"f4cc3f4e-9300-479c-b129-c0f12eed77f1","Type":"ContainerStarted","Data":"c550af9362c07ad83830c617194ee7f717f264d6a82652464e94259da224cbcd"} Oct 31 21:46:23.575324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:23.572101 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" event={"ID":"583ea135-1351-4fd2-a308-42bf4fab268e","Type":"ContainerStarted","Data":"9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39"} Oct 31 21:46:23.623814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:23.623759 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podStartSLOduration=1.623743782 podStartE2EDuration="1.623743782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:23.622061816 +0000 UTC m=+1823.747169043" watchObservedRunningTime="2026-10-31 21:46:23.623743782 +0000 UTC m=+1823.748851091" Oct 31 21:46:23.666712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:23.666663 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podStartSLOduration=2.666649779 podStartE2EDuration="2.666649779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:23.666176073 +0000 UTC m=+1823.791283318" watchObservedRunningTime="2026-10-31 21:46:23.666649779 +0000 UTC m=+1823.791757009" Oct 31 21:46:24.159793 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:24.159757 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:24.161812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:24.161787 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:46:24.161956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:24.161833 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 21:46:24.455568 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:24.455472 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:24.461269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:24.461243 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:24.638169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:24.629200 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:24.638169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:24.636581 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:46:25.160118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.160090 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:46:25.160183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.160162 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 21:46:25.436973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.436952 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:46:25.445146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445125 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-credential-socket\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445193 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-certs\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445271 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445253 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istiod-ca-cert\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445361 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-data\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445412 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-envoy\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445458 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-token\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-token\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445460 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-credential-socket" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:46:25.445592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445486 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-certs" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:46:25.445592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445503 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-kube-api-access-4lfct\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-kube-api-access-4lfct\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445547 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-podinfo\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445586 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-socket\") pod \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\" (UID: \"c9df0bfc-50ed-49c8-98f2-d0803059d97d\") " Oct 31 21:46:25.445792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445594 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-data" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:46:25.445792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.445677 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istiod-ca-cert" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:25.446080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.446055 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-socket" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:46:25.446201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.446185 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.446270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.446209 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.446270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.446223 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.446270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.446235 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.446270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.446248 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.447686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.447659 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-podinfo" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:46:25.447780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.447721 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-kube-api-access-4lfct" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "kube-api-access-4lfct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:25.447780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.447741 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-token" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:25.447852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.447810 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-envoy" pod "c9df0bfc-50ed-49c8-98f2-d0803059d97d" (UID: "c9df0bfc-50ed-49c8-98f2-d0803059d97d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:46:25.547960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.547926 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.547960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.547956 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.547960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.547965 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lfct\" (UniqueName: \"kubernetes.io/projected/c9df0bfc-50ed-49c8-98f2-d0803059d97d-kube-api-access-4lfct\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.548177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.547976 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9df0bfc-50ed-49c8-98f2-d0803059d97d-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:25.632167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.632133 4294 generic.go:356] "Generic (PLEG): container finished" podID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerID="8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911" exitCode=0 Oct 31 21:46:25.632387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.632191 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" Oct 31 21:46:25.632387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.632215 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" event={"ID":"c9df0bfc-50ed-49c8-98f2-d0803059d97d","Type":"ContainerDied","Data":"8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911"} Oct 31 21:46:25.632387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.632272 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm" event={"ID":"c9df0bfc-50ed-49c8-98f2-d0803059d97d","Type":"ContainerDied","Data":"ad80be5928b7ee8b096ca29d431cb692b65357ae6f0a1d63933b8edc1412e0f7"} Oct 31 21:46:25.632387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.632295 4294 scope.go:122] "RemoveContainer" containerID="8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911" Oct 31 21:46:25.645168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.645151 4294 scope.go:122] "RemoveContainer" containerID="9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26" Oct 31 21:46:25.661368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.661344 4294 scope.go:122] "RemoveContainer" containerID="8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911" Oct 31 21:46:25.661657 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:46:25.661636 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911\": container with ID starting with 8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911 not found: ID does not exist" containerID="8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911" Oct 31 21:46:25.661740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.661669 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911"} err="failed to get container status \"8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911\": rpc error: code = NotFound desc = could not find container \"8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911\": container with ID starting with 8769bc34348d04608993b8453eda660706cb593a51b02ea5091c8e8e8022e911 not found: ID does not exist" Oct 31 21:46:25.661740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.661692 4294 scope.go:122] "RemoveContainer" containerID="9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26" Oct 31 21:46:25.661986 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:46:25.661967 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26\": container with ID starting with 9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26 not found: ID does not exist" containerID="9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26" Oct 31 21:46:25.662044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.661996 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26"} err="failed to get container status \"9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26\": rpc error: code = NotFound desc = could not find container \"9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26\": container with ID starting with 9a29742574c7a293a2bae9d483f9ca6279dfd17785d0f7ba523fb3e62269fc26 not found: ID does not exist" Oct 31 21:46:25.699041 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.699002 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm"] Oct 31 21:46:25.702584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:25.702559 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-09f39e-openshift-default-d7f9cc8c7-596fm"] Oct 31 21:46:26.004417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:26.004385 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" path="/var/lib/kubelet/pods/c9df0bfc-50ed-49c8-98f2-d0803059d97d/volumes" Oct 31 21:46:26.160291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:26.160259 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:46:26.160474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:26.160364 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 21:46:27.160864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:27.160834 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:46:27.161244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:27.160907 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 21:46:28.160796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:28.160753 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:46:28.161015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:28.160847 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 21:46:29.160620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:29.160581 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 21:46:29.160822 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:29.160639 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 21:46:30.165013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:30.164980 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:30.165437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:30.165283 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:30.166046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:30.166029 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:46:31.310922 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:46:31.310889 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:46:34.435552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.435513 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g"] Oct 31 21:46:34.436059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.436052 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" Oct 31 21:46:34.436112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.436063 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" Oct 31 21:46:34.436112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.436075 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" Oct 31 21:46:34.436112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.436080 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" Oct 31 21:46:34.436223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.436175 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" Oct 31 21:46:34.436223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.436184 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9df0bfc-50ed-49c8-98f2-d0803059d97d" containerName="istio-proxy" Oct 31 21:46:34.440553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.440536 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.457743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.457712 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g"] Oct 31 21:46:34.465364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.465339 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:46:34.465481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.465426 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:46:34.486494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.486467 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g252\" (UniqueName: \"kubernetes.io/projected/093719aa-559e-4a7d-a2c8-59386676956e-kube-api-access-8g252\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.486608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.486520 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.486902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.486877 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.588475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.588436 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.588658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.588521 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g252\" (UniqueName: \"kubernetes.io/projected/093719aa-559e-4a7d-a2c8-59386676956e-kube-api-access-8g252\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.588658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.588554 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.589112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.589086 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.590840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.590816 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.611770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.611748 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g252\" (UniqueName: \"kubernetes.io/projected/093719aa-559e-4a7d-a2c8-59386676956e-kube-api-access-8g252\") pod \"maas-api-revoke-sub-b10fe77b-97d2g\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:34.753872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:34.753843 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:35.062863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:35.062824 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g"] Oct 31 21:46:35.063523 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:35.063500 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093719aa_559e_4a7d_a2c8_59386676956e.slice/crio-96dcbfded6367c790b727a7562fe5dc0568bea097b9d05cbb7d6cf851c0494a9 WatchSource:0}: Error finding container 96dcbfded6367c790b727a7562fe5dc0568bea097b9d05cbb7d6cf851c0494a9: Status 404 returned error can't find the container with id 96dcbfded6367c790b727a7562fe5dc0568bea097b9d05cbb7d6cf851c0494a9 Oct 31 21:46:35.711977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:35.711891 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" event={"ID":"093719aa-559e-4a7d-a2c8-59386676956e","Type":"ContainerStarted","Data":"5497b9698038b76bf3faff4a8b6cb4a299483cce5b2516de166b67bc13861a85"} Oct 31 21:46:35.711977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:35.711947 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" event={"ID":"093719aa-559e-4a7d-a2c8-59386676956e","Type":"ContainerStarted","Data":"96dcbfded6367c790b727a7562fe5dc0568bea097b9d05cbb7d6cf851c0494a9"} Oct 31 21:46:35.757761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:35.757710 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" podStartSLOduration=1.757696874 podStartE2EDuration="1.757696874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:35.753918975 +0000 UTC m=+1835.879026216" watchObservedRunningTime="2026-10-31 21:46:35.757696874 +0000 UTC m=+1835.882804104" Oct 31 21:46:36.036543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.036503 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww"] Oct 31 21:46:36.043952 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.043888 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-432f3459-bkwww"] Oct 31 21:46:36.276725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.259187 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7"] Oct 31 21:46:36.276725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.259512 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" podUID="d00c8eec-1cb8-447a-9c8a-e4671aabe194" containerName="maas-api" containerID="cri-o://e692c643163ec91d7784beef46677b922facf436386ad1613d5445b2a020b6eb" gracePeriod=30 Oct 31 21:46:36.746091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.745929 4294 generic.go:356] "Generic (PLEG): container finished" podID="d00c8eec-1cb8-447a-9c8a-e4671aabe194" containerID="e692c643163ec91d7784beef46677b922facf436386ad1613d5445b2a020b6eb" exitCode=0 Oct 31 21:46:36.746091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.746020 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" event={"ID":"d00c8eec-1cb8-447a-9c8a-e4671aabe194","Type":"ContainerDied","Data":"e692c643163ec91d7784beef46677b922facf436386ad1613d5445b2a020b6eb"} Oct 31 21:46:36.749110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.748603 4294 generic.go:356] "Generic (PLEG): container finished" podID="093719aa-559e-4a7d-a2c8-59386676956e" containerID="5497b9698038b76bf3faff4a8b6cb4a299483cce5b2516de166b67bc13861a85" exitCode=0 Oct 31 21:46:36.749110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.748735 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" event={"ID":"093719aa-559e-4a7d-a2c8-59386676956e","Type":"ContainerDied","Data":"5497b9698038b76bf3faff4a8b6cb4a299483cce5b2516de166b67bc13861a85"} Oct 31 21:46:36.951328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:36.916059 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2"] Oct 31 21:46:37.105403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.079658 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2"] Oct 31 21:46:37.127416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.083466 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222366 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222429 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222456 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408faa8a-75a3-4f10-babf-6de042231a88-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222530 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222594 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222703 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhzh\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-kube-api-access-vdhzh\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222800 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222842 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408faa8a-75a3-4f10-babf-6de042231a88-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.223325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.222878 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325139 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325204 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408faa8a-75a3-4f10-babf-6de042231a88-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325237 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325334 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325382 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325412 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408faa8a-75a3-4f10-babf-6de042231a88-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325490 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325556 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.325636 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhzh\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-kube-api-access-vdhzh\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.326433 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.327164 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408faa8a-75a3-4f10-babf-6de042231a88-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.348555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.327443 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.365600 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.365517 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.366025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.366000 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27"] Oct 31 21:46:37.366479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.366457 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.432396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.432355 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-448b01-lfd27"] Oct 31 21:46:37.456335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.445923 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408faa8a-75a3-4f10-babf-6de042231a88-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.503342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.480604 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.503342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.481268 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhzh\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-kube-api-access-vdhzh\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.575329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.563185 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:37.597704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:37.597666 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:38.074723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.074655 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881bbd23-8801-42cc-8d8c-1f57746d952f" path="/var/lib/kubelet/pods/881bbd23-8801-42cc-8d8c-1f57746d952f/volumes" Oct 31 21:46:38.079825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.077950 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb770b44-cc79-4ed3-b534-d0d11e867eee" path="/var/lib/kubelet/pods/cb770b44-cc79-4ed3-b534-d0d11e867eee/volumes" Oct 31 21:46:38.080727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.080704 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:46:38.166865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.166828 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-metrics-certs\") pod \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " Oct 31 21:46:38.167019 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.166896 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d00c8eec-1cb8-447a-9c8a-e4671aabe194-kube-api-access-v4mjc\" (UniqueName: \"kubernetes.io/projected/d00c8eec-1cb8-447a-9c8a-e4671aabe194-kube-api-access-v4mjc\") pod \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " Oct 31 21:46:38.167123 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.167096 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-maas-api-tls\") pod \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " Oct 31 21:46:38.167191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.167144 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d00c8eec-1cb8-447a-9c8a-e4671aabe194-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c8eec-1cb8-447a-9c8a-e4671aabe194-openshift-service-ca\") pod \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\" (UID: \"d00c8eec-1cb8-447a-9c8a-e4671aabe194\") " Oct 31 21:46:38.169323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.169278 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00c8eec-1cb8-447a-9c8a-e4671aabe194-openshift-service-ca" pod "d00c8eec-1cb8-447a-9c8a-e4671aabe194" (UID: "d00c8eec-1cb8-447a-9c8a-e4671aabe194"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:38.176632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.176588 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00c8eec-1cb8-447a-9c8a-e4671aabe194-kube-api-access-v4mjc" pod "d00c8eec-1cb8-447a-9c8a-e4671aabe194" (UID: "d00c8eec-1cb8-447a-9c8a-e4671aabe194"). InnerVolumeSpecName "kube-api-access-v4mjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:38.178758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.178720 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-maas-api-tls" pod "d00c8eec-1cb8-447a-9c8a-e4671aabe194" (UID: "d00c8eec-1cb8-447a-9c8a-e4671aabe194"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:38.193878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.192600 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-metrics-certs" pod "d00c8eec-1cb8-447a-9c8a-e4671aabe194" (UID: "d00c8eec-1cb8-447a-9c8a-e4671aabe194"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:38.269323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.269143 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:38.269323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.269189 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00c8eec-1cb8-447a-9c8a-e4671aabe194-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:38.269323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.269205 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00c8eec-1cb8-447a-9c8a-e4671aabe194-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:38.269323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.269219 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4mjc\" (UniqueName: \"kubernetes.io/projected/d00c8eec-1cb8-447a-9c8a-e4671aabe194-kube-api-access-v4mjc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:38.360148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.360122 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:38.428948 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:38.428915 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408faa8a_75a3_4f10_babf_6de042231a88.slice/crio-71b5a2a9dc706196caa28a7fa050eb8913145fb5c54f0d339580fd792c0c4b8e WatchSource:0}: Error finding container 71b5a2a9dc706196caa28a7fa050eb8913145fb5c54f0d339580fd792c0c4b8e: Status 404 returned error can't find the container with id 71b5a2a9dc706196caa28a7fa050eb8913145fb5c54f0d339580fd792c0c4b8e Oct 31 21:46:38.429111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.429039 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2"] Oct 31 21:46:38.431723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.431688 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:38.431843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.431757 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:38.431843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.431795 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:46:38.471550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.471518 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093719aa-559e-4a7d-a2c8-59386676956e-kube-api-access-8g252\" (UniqueName: \"kubernetes.io/projected/093719aa-559e-4a7d-a2c8-59386676956e-kube-api-access-8g252\") pod \"093719aa-559e-4a7d-a2c8-59386676956e\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " Oct 31 21:46:38.471700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.471663 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-xks-ca\") pod \"093719aa-559e-4a7d-a2c8-59386676956e\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " Oct 31 21:46:38.471752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.471713 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-openshift-ca\") pod \"093719aa-559e-4a7d-a2c8-59386676956e\" (UID: \"093719aa-559e-4a7d-a2c8-59386676956e\") " Oct 31 21:46:38.472064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.472039 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-openshift-ca" pod "093719aa-559e-4a7d-a2c8-59386676956e" (UID: "093719aa-559e-4a7d-a2c8-59386676956e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:38.472379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.472359 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:38.473891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.473654 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093719aa-559e-4a7d-a2c8-59386676956e-kube-api-access-8g252" pod "093719aa-559e-4a7d-a2c8-59386676956e" (UID: "093719aa-559e-4a7d-a2c8-59386676956e"). InnerVolumeSpecName "kube-api-access-8g252". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:38.473891 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.473773 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-xks-ca" pod "093719aa-559e-4a7d-a2c8-59386676956e" (UID: "093719aa-559e-4a7d-a2c8-59386676956e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:38.573268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.573237 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g252\" (UniqueName: \"kubernetes.io/projected/093719aa-559e-4a7d-a2c8-59386676956e-kube-api-access-8g252\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:38.573268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.573268 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/093719aa-559e-4a7d-a2c8-59386676956e-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:38.902508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.902477 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" event={"ID":"d00c8eec-1cb8-447a-9c8a-e4671aabe194","Type":"ContainerDied","Data":"2c3c7322cacff025f7d574e444e6f1c8bacaef14f3ccfbad714574fe60d3363d"} Oct 31 21:46:38.902710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.902539 4294 scope.go:122] "RemoveContainer" containerID="e692c643163ec91d7784beef46677b922facf436386ad1613d5445b2a020b6eb" Oct 31 21:46:38.902710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.902483 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" Oct 31 21:46:38.904039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.904016 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" Oct 31 21:46:38.904153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.904041 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g" event={"ID":"093719aa-559e-4a7d-a2c8-59386676956e","Type":"ContainerDied","Data":"96dcbfded6367c790b727a7562fe5dc0568bea097b9d05cbb7d6cf851c0494a9"} Oct 31 21:46:38.904153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.904084 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96dcbfded6367c790b727a7562fe5dc0568bea097b9d05cbb7d6cf851c0494a9" Oct 31 21:46:38.905979 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.905945 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" event={"ID":"408faa8a-75a3-4f10-babf-6de042231a88","Type":"ContainerStarted","Data":"59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6"} Oct 31 21:46:38.906064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.905994 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" event={"ID":"408faa8a-75a3-4f10-babf-6de042231a88","Type":"ContainerStarted","Data":"71b5a2a9dc706196caa28a7fa050eb8913145fb5c54f0d339580fd792c0c4b8e"} Oct 31 21:46:38.980851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:38.980804 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podStartSLOduration=2.980788168 podStartE2EDuration="2.980788168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:38.976692499 +0000 UTC m=+1839.101799732" watchObservedRunningTime="2026-10-31 21:46:38.980788168 +0000 UTC m=+1839.105895397" Oct 31 21:46:39.014563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.014529 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7"] Oct 31 21:46:39.018831 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.018802 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7"] Oct 31 21:46:39.205024 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.204947 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz"] Oct 31 21:46:39.205510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.205496 4294 cpu_manager.go:414] "containerMap: removing container" podUID="093719aa-559e-4a7d-a2c8-59386676956e" containerName="revoke-keys" Oct 31 21:46:39.205510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.205511 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093719aa-559e-4a7d-a2c8-59386676956e" containerName="revoke-keys" Oct 31 21:46:39.205594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.205526 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d00c8eec-1cb8-447a-9c8a-e4671aabe194" containerName="maas-api" Oct 31 21:46:39.205594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.205531 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d00c8eec-1cb8-447a-9c8a-e4671aabe194" containerName="maas-api" Oct 31 21:46:39.205656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.205632 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="093719aa-559e-4a7d-a2c8-59386676956e" containerName="revoke-keys" Oct 31 21:46:39.205656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.205641 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d00c8eec-1cb8-447a-9c8a-e4671aabe194" containerName="maas-api" Oct 31 21:46:39.210192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.210174 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.225182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.225156 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:46:39.225317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.225230 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:46:39.228515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.228488 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz"] Oct 31 21:46:39.282439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.282406 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.282599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.282475 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmk99\" (UniqueName: \"kubernetes.io/projected/76e4c87b-7894-4980-9a06-7f7087adff5c-kube-api-access-jmk99\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.282599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.282515 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.383743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.383711 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.383899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.383764 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmk99\" (UniqueName: \"kubernetes.io/projected/76e4c87b-7894-4980-9a06-7f7087adff5c-kube-api-access-jmk99\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.383899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.383791 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.384372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.384351 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.386080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.386059 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.409758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.409723 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmk99\" (UniqueName: \"kubernetes.io/projected/76e4c87b-7894-4980-9a06-7f7087adff5c-kube-api-access-jmk99\") pod \"maas-api-revoke-sub-42f3e9b9-v4tlz\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.524855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.524811 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:39.605631 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.605597 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:39.607203 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.607180 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 21:46:39.607275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.607231 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 21:46:39.863089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.863061 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz"] Oct 31 21:46:39.934645 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:39.934609 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" event={"ID":"76e4c87b-7894-4980-9a06-7f7087adff5c","Type":"ContainerStarted","Data":"eae3c39314b14deab6ee2a8cc6ecc99ab7dd2873727f1045bab39014d4434058"} Oct 31 21:46:40.006531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:40.006494 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d00c8eec-1cb8-447a-9c8a-e4671aabe194" path="/var/lib/kubelet/pods/d00c8eec-1cb8-447a-9c8a-e4671aabe194/volumes" Oct 31 21:46:40.276612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:40.276108 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-448b01-6789d7f748-mlvz7" podUID="d00c8eec-1cb8-447a-9c8a-e4671aabe194" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:46:40.606595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:40.606560 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 21:46:40.606778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:40.606621 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 21:46:40.947209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:40.947097 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" event={"ID":"76e4c87b-7894-4980-9a06-7f7087adff5c","Type":"ContainerStarted","Data":"cc27606128da81628d0f6484775ba6ca3befb62853f8a4d858464f28a30891e5"} Oct 31 21:46:40.998615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:40.997896 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" podStartSLOduration=1.997877673 podStartE2EDuration="1.997877673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:40.991724952 +0000 UTC m=+1841.116832196" watchObservedRunningTime="2026-10-31 21:46:40.997877673 +0000 UTC m=+1841.122984906" Oct 31 21:46:41.606406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:41.606368 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 21:46:41.606811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:41.606424 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 21:46:41.955470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:41.955391 4294 generic.go:356] "Generic (PLEG): container finished" podID="76e4c87b-7894-4980-9a06-7f7087adff5c" containerID="cc27606128da81628d0f6484775ba6ca3befb62853f8a4d858464f28a30891e5" exitCode=0 Oct 31 21:46:41.955635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:41.955470 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" event={"ID":"76e4c87b-7894-4980-9a06-7f7087adff5c","Type":"ContainerDied","Data":"cc27606128da81628d0f6484775ba6ca3befb62853f8a4d858464f28a30891e5"} Oct 31 21:46:42.606196 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:42.606157 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 21:46:42.606402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:42.606216 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 21:46:43.163974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.163949 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:43.267214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.267172 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-openshift-ca\") pod \"76e4c87b-7894-4980-9a06-7f7087adff5c\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " Oct 31 21:46:43.267402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.267254 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76e4c87b-7894-4980-9a06-7f7087adff5c-kube-api-access-jmk99\" (UniqueName: \"kubernetes.io/projected/76e4c87b-7894-4980-9a06-7f7087adff5c-kube-api-access-jmk99\") pod \"76e4c87b-7894-4980-9a06-7f7087adff5c\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " Oct 31 21:46:43.267402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.267289 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-xks-ca\") pod \"76e4c87b-7894-4980-9a06-7f7087adff5c\" (UID: \"76e4c87b-7894-4980-9a06-7f7087adff5c\") " Oct 31 21:46:43.268916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.268892 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-openshift-ca" pod "76e4c87b-7894-4980-9a06-7f7087adff5c" (UID: "76e4c87b-7894-4980-9a06-7f7087adff5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:43.270447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.270420 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-xks-ca" pod "76e4c87b-7894-4980-9a06-7f7087adff5c" (UID: "76e4c87b-7894-4980-9a06-7f7087adff5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:43.270852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.270825 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76e4c87b-7894-4980-9a06-7f7087adff5c-kube-api-access-jmk99" pod "76e4c87b-7894-4980-9a06-7f7087adff5c" (UID: "76e4c87b-7894-4980-9a06-7f7087adff5c"). InnerVolumeSpecName "kube-api-access-jmk99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:43.369956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.369922 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:43.369956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.369949 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmk99\" (UniqueName: \"kubernetes.io/projected/76e4c87b-7894-4980-9a06-7f7087adff5c-kube-api-access-jmk99\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:43.369956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.369961 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76e4c87b-7894-4980-9a06-7f7087adff5c-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:43.605858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.605766 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 21:46:43.605858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.605825 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 21:46:43.998839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:43.998798 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" Oct 31 21:46:44.016120 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.016078 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz" event={"ID":"76e4c87b-7894-4980-9a06-7f7087adff5c","Type":"ContainerDied","Data":"eae3c39314b14deab6ee2a8cc6ecc99ab7dd2873727f1045bab39014d4434058"} Oct 31 21:46:44.016339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.016139 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eae3c39314b14deab6ee2a8cc6ecc99ab7dd2873727f1045bab39014d4434058" Oct 31 21:46:44.605824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.605784 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 21:46:44.606232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.605839 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 21:46:44.929184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.929090 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh"] Oct 31 21:46:44.929712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.929694 4294 cpu_manager.go:414] "containerMap: removing container" podUID="76e4c87b-7894-4980-9a06-7f7087adff5c" containerName="revoke-keys" Oct 31 21:46:44.929712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.929709 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76e4c87b-7894-4980-9a06-7f7087adff5c" containerName="revoke-keys" Oct 31 21:46:44.929902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.929818 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="76e4c87b-7894-4980-9a06-7f7087adff5c" containerName="revoke-keys" Oct 31 21:46:44.933518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.933499 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:44.953126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:44.953095 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh"] Oct 31 21:46:45.004869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.004841 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:46:45.005047 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.004883 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:46:45.019290 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.019260 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pth\" (UniqueName: \"kubernetes.io/projected/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-kube-api-access-x4pth\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.019446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.019427 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.019605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.019582 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.121361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.121282 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.121726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.121703 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.122656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.122538 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.122805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.122701 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pth\" (UniqueName: \"kubernetes.io/projected/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-kube-api-access-x4pth\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.124749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.124729 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.149594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.149561 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pth\" (UniqueName: \"kubernetes.io/projected/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-kube-api-access-x4pth\") pod \"maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.250136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.250098 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:45.561996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:45.561964 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh"] Oct 31 21:46:45.563645 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:45.563619 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aff6a0b_fbf2_42a9_ba8c_84a255cecdbf.slice/crio-041ada2d9b2f0c84f56505dbee25737c72dd1ee7fe8d9798ba9baf1970f3fc6f WatchSource:0}: Error finding container 041ada2d9b2f0c84f56505dbee25737c72dd1ee7fe8d9798ba9baf1970f3fc6f: Status 404 returned error can't find the container with id 041ada2d9b2f0c84f56505dbee25737c72dd1ee7fe8d9798ba9baf1970f3fc6f Oct 31 21:46:46.041866 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.041806 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx"] Oct 31 21:46:46.042986 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.042949 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" event={"ID":"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf","Type":"ContainerStarted","Data":"6124eeeefb2c8e69703a3ff58d5eeaa3752a306dcd0c533d640b23edd4635e47"} Oct 31 21:46:46.043160 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.043015 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" event={"ID":"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf","Type":"ContainerStarted","Data":"041ada2d9b2f0c84f56505dbee25737c72dd1ee7fe8d9798ba9baf1970f3fc6f"} Oct 31 21:46:46.046478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.046452 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54dbb230-76xcx"] Oct 31 21:46:46.075230 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.075193 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:46.075772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.075748 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:46.076622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.076603 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:46:46.088327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:46.086295 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" podStartSLOduration=2.08628269 podStartE2EDuration="2.08628269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:46:46.083164913 +0000 UTC m=+1846.208272126" watchObservedRunningTime="2026-10-31 21:46:46.08628269 +0000 UTC m=+1846.211389920" Oct 31 21:46:47.052360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:47.052294 4294 generic.go:356] "Generic (PLEG): container finished" podID="0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" containerID="6124eeeefb2c8e69703a3ff58d5eeaa3752a306dcd0c533d640b23edd4635e47" exitCode=0 Oct 31 21:46:47.052360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:47.052341 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" event={"ID":"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf","Type":"ContainerDied","Data":"6124eeeefb2c8e69703a3ff58d5eeaa3752a306dcd0c533d640b23edd4635e47"} Oct 31 21:46:47.993488 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:47.993451 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbdbe909-ff3e-4564-acdd-56592436dddf" path="/var/lib/kubelet/pods/fbdbe909-ff3e-4564-acdd-56592436dddf/volumes" Oct 31 21:46:48.210829 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.210798 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:48.375202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.375109 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-openshift-ca\") pod \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " Oct 31 21:46:48.375202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.375169 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-kube-api-access-x4pth\" (UniqueName: \"kubernetes.io/projected/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-kube-api-access-x4pth\") pod \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " Oct 31 21:46:48.375202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.375197 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-xks-ca\") pod \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\" (UID: \"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf\") " Oct 31 21:46:48.375514 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.375474 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-openshift-ca" pod "0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" (UID: "0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:48.375728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.375711 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:48.377369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.377343 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-xks-ca" pod "0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" (UID: "0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:48.377476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.377392 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-kube-api-access-x4pth" pod "0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" (UID: "0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf"). InnerVolumeSpecName "kube-api-access-x4pth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:48.477500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.477466 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4pth\" (UniqueName: \"kubernetes.io/projected/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-kube-api-access-x4pth\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:48.477500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:48.477499 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:49.067866 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:49.067842 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" Oct 31 21:46:49.068043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:49.067962 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh" event={"ID":"0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf","Type":"ContainerDied","Data":"041ada2d9b2f0c84f56505dbee25737c72dd1ee7fe8d9798ba9baf1970f3fc6f"} Oct 31 21:46:49.068043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:49.068004 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="041ada2d9b2f0c84f56505dbee25737c72dd1ee7fe8d9798ba9baf1970f3fc6f" Oct 31 21:46:53.371187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.371155 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65"] Oct 31 21:46:53.371721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.371708 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" containerName="revoke-keys" Oct 31 21:46:53.371776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.371723 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" containerName="revoke-keys" Oct 31 21:46:53.371825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.371816 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" containerName="revoke-keys" Oct 31 21:46:53.375396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.375379 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.394384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.394357 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65"] Oct 31 21:46:53.402290 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.402255 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:46:53.402407 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.402350 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:46:53.450823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.450789 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.450974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.450876 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mc97\" (UniqueName: \"kubernetes.io/projected/71ffeef8-93b1-4b63-922a-0f9b54b7b566-kube-api-access-6mc97\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.451045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.451024 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.552582 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.552551 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.552752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.552663 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.552752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.552708 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mc97\" (UniqueName: \"kubernetes.io/projected/71ffeef8-93b1-4b63-922a-0f9b54b7b566-kube-api-access-6mc97\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.553288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.553264 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.554939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.554916 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.575899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.575878 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mc97\" (UniqueName: \"kubernetes.io/projected/71ffeef8-93b1-4b63-922a-0f9b54b7b566-kube-api-access-6mc97\") pod \"maas-api-revoke-sub-d5ad7956-54s65\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.688460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.688381 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:53.999109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:53.999086 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65"] Oct 31 21:46:54.000432 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:46:54.000403 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ffeef8_93b1_4b63_922a_0f9b54b7b566.slice/crio-e549acb8202d2aa7d3d55adefba7cef049feb217370036045fd0c3a01b35233d WatchSource:0}: Error finding container e549acb8202d2aa7d3d55adefba7cef049feb217370036045fd0c3a01b35233d: Status 404 returned error can't find the container with id e549acb8202d2aa7d3d55adefba7cef049feb217370036045fd0c3a01b35233d Oct 31 21:46:54.105551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:54.105517 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" event={"ID":"71ffeef8-93b1-4b63-922a-0f9b54b7b566","Type":"ContainerStarted","Data":"e549acb8202d2aa7d3d55adefba7cef049feb217370036045fd0c3a01b35233d"} Oct 31 21:46:55.115592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:55.115560 4294 generic.go:356] "Generic (PLEG): container finished" podID="71ffeef8-93b1-4b63-922a-0f9b54b7b566" containerID="e7941b04b3a03919f3f880b74c77fabc87234e98af57e0d3a7da8a97ee94ea86" exitCode=0 Oct 31 21:46:55.115976 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:55.115640 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" event={"ID":"71ffeef8-93b1-4b63-922a-0f9b54b7b566","Type":"ContainerDied","Data":"e7941b04b3a03919f3f880b74c77fabc87234e98af57e0d3a7da8a97ee94ea86"} Oct 31 21:46:55.472752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:55.472719 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr"] Oct 31 21:46:55.505928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:55.505890 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw"] Oct 31 21:46:56.199137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.199101 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp"] Oct 31 21:46:56.203004 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.202975 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-80542b-f2ttp"] Oct 31 21:46:56.351330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.350053 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:56.407969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.407882 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-xks-ca\") pod \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " Oct 31 21:46:56.408141 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.407974 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-openshift-ca\") pod \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " Oct 31 21:46:56.408141 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.408037 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ffeef8-93b1-4b63-922a-0f9b54b7b566-kube-api-access-6mc97\" (UniqueName: \"kubernetes.io/projected/71ffeef8-93b1-4b63-922a-0f9b54b7b566-kube-api-access-6mc97\") pod \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\" (UID: \"71ffeef8-93b1-4b63-922a-0f9b54b7b566\") " Oct 31 21:46:56.408463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.408431 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-openshift-ca" pod "71ffeef8-93b1-4b63-922a-0f9b54b7b566" (UID: "71ffeef8-93b1-4b63-922a-0f9b54b7b566"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:46:56.408833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.408805 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:56.410429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.410402 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ffeef8-93b1-4b63-922a-0f9b54b7b566-kube-api-access-6mc97" pod "71ffeef8-93b1-4b63-922a-0f9b54b7b566" (UID: "71ffeef8-93b1-4b63-922a-0f9b54b7b566"). InnerVolumeSpecName "kube-api-access-6mc97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:46:56.411432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.411409 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-xks-ca" pod "71ffeef8-93b1-4b63-922a-0f9b54b7b566" (UID: "71ffeef8-93b1-4b63-922a-0f9b54b7b566"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:46:56.510055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.510011 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ffeef8-93b1-4b63-922a-0f9b54b7b566-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:56.510055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:56.510045 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mc97\" (UniqueName: \"kubernetes.io/projected/71ffeef8-93b1-4b63-922a-0f9b54b7b566-kube-api-access-6mc97\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:46:57.130501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:57.130467 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" event={"ID":"71ffeef8-93b1-4b63-922a-0f9b54b7b566","Type":"ContainerDied","Data":"e549acb8202d2aa7d3d55adefba7cef049feb217370036045fd0c3a01b35233d"} Oct 31 21:46:57.130671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:57.130504 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65" Oct 31 21:46:57.130671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:57.130514 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e549acb8202d2aa7d3d55adefba7cef049feb217370036045fd0c3a01b35233d" Oct 31 21:46:57.985434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:46:57.985401 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1b1051-c8bd-4829-8b59-a999a9807a0b" path="/var/lib/kubelet/pods/ea1b1051-c8bd-4829-8b59-a999a9807a0b/volumes" Oct 31 21:47:00.473819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:00.473751 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" podUID="6a940e08-2935-495c-896a-74a979b821c6" containerName="payload-processing" containerID="cri-o://19040186a54a7f35cd60a85873533b2500fa2c62f383a581e98618ebe050ab53" gracePeriod=25 Oct 31 21:47:00.506499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:00.506439 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" podUID="e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" containerName="payload-pre-processing" containerID="cri-o://f8606f7ec406c6d062dad19925eeaae6fed2aa38e006978b84b0b262be01e0b6" gracePeriod=25 Oct 31 21:47:00.656443 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:00.656275 4294 scope.go:122] "RemoveContainer" containerID="98fe6aaaf681729ca5c87b21a3d4f7c19d107b72eda264db28273e3e676ab3ce" Oct 31 21:47:00.693922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:00.693521 4294 scope.go:122] "RemoveContainer" containerID="01967b1f7621b2ec3ecff9ef88336e540f75eb17f35526b825cbf12aef98f278" Oct 31 21:47:01.685103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.682659 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5"] Oct 31 21:47:01.709551 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:47:01.700806 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:47:01.709551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.701060 4294 generic.go:356] "Generic (PLEG): container finished" podID="e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" containerID="f8606f7ec406c6d062dad19925eeaae6fed2aa38e006978b84b0b262be01e0b6" exitCode=0 Oct 31 21:47:01.709551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.701148 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" event={"ID":"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd","Type":"ContainerDied","Data":"f8606f7ec406c6d062dad19925eeaae6fed2aa38e006978b84b0b262be01e0b6"} Oct 31 21:47:01.724507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.723954 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:47:01.763326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.757958 4294 generic.go:356] "Generic (PLEG): container finished" podID="6a940e08-2935-495c-896a-74a979b821c6" containerID="19040186a54a7f35cd60a85873533b2500fa2c62f383a581e98618ebe050ab53" exitCode=0 Oct 31 21:47:01.763326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.758151 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" event={"ID":"6a940e08-2935-495c-896a-74a979b821c6","Type":"ContainerDied","Data":"19040186a54a7f35cd60a85873533b2500fa2c62f383a581e98618ebe050ab53"} Oct 31 21:47:01.763326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.758219 4294 scope.go:122] "RemoveContainer" containerID="19040186a54a7f35cd60a85873533b2500fa2c62f383a581e98618ebe050ab53" Oct 31 21:47:01.833355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.825594 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a940e08-2935-495c-896a-74a979b821c6-kube-api-access-4plsz\" (UniqueName: \"kubernetes.io/projected/6a940e08-2935-495c-896a-74a979b821c6-kube-api-access-4plsz\") pod \"6a940e08-2935-495c-896a-74a979b821c6\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " Oct 31 21:47:01.833355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.825680 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a940e08-2935-495c-896a-74a979b821c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a940e08-2935-495c-896a-74a979b821c6-plugins-config-volume\") pod \"6a940e08-2935-495c-896a-74a979b821c6\" (UID: \"6a940e08-2935-495c-896a-74a979b821c6\") " Oct 31 21:47:01.841322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.841047 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a940e08-2935-495c-896a-74a979b821c6-plugins-config-volume" pod "6a940e08-2935-495c-896a-74a979b821c6" (UID: "6a940e08-2935-495c-896a-74a979b821c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:01.869728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.869477 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a940e08-2935-495c-896a-74a979b821c6-kube-api-access-4plsz" pod "6a940e08-2935-495c-896a-74a979b821c6" (UID: "6a940e08-2935-495c-896a-74a979b821c6"). InnerVolumeSpecName "kube-api-access-4plsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:01.898328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.882767 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:47:01.960401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.960364 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plsz\" (UniqueName: \"kubernetes.io/projected/6a940e08-2935-495c-896a-74a979b821c6-kube-api-access-4plsz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:01.960628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:01.960609 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a940e08-2935-495c-896a-74a979b821c6-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:02.161530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.127769 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-plugins-config-volume\") pod \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " Oct 31 21:47:02.161530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.127972 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-kube-api-access-m8qzw\" (UniqueName: \"kubernetes.io/projected/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-kube-api-access-m8qzw\") pod \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\" (UID: \"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd\") " Oct 31 21:47:02.161530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.154901 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-plugins-config-volume" pod "e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" (UID: "e90c5062-d9a4-4f71-ae01-ea8f863ef1cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:02.253327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.233336 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:02.260551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.260510 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-kube-api-access-m8qzw" pod "e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" (UID: "e90c5062-d9a4-4f71-ae01-ea8f863ef1cd"). InnerVolumeSpecName "kube-api-access-m8qzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:02.336826 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.336356 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8qzw\" (UniqueName: \"kubernetes.io/projected/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd-kube-api-access-m8qzw\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:02.663790 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.663719 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded" Oct 31 21:47:02.782666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.782612 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" event={"ID":"e90c5062-d9a4-4f71-ae01-ea8f863ef1cd","Type":"ContainerDied","Data":"20b214407eb57a4c3bb14d7b4c3a92afcd8517234d962362e52633aec8ef7237"} Oct 31 21:47:02.783286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.783248 4294 scope.go:122] "RemoveContainer" containerID="f8606f7ec406c6d062dad19925eeaae6fed2aa38e006978b84b0b262be01e0b6" Oct 31 21:47:02.783625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.783609 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw" Oct 31 21:47:02.811670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.811494 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded" Oct 31 21:47:02.893387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.893143 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" event={"ID":"6a940e08-2935-495c-896a-74a979b821c6","Type":"ContainerDied","Data":"87ca031fedf8016848724c200294bb336cbce6fba559297b545444a3cd653f53"} Oct 31 21:47:02.893491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.893467 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr" Oct 31 21:47:02.912326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.909571 4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 31 21:47:02.991408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:02.980358 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw"] Oct 31 21:47:03.005887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.005681 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:47:03.044329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.042072 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-1f523a-f86c9c547-8vkdw"] Oct 31 21:47:03.060326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.057418 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr"] Oct 31 21:47:03.087326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.086413 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-1f523a-867bb8df5c-szvlr"] Oct 31 21:47:03.126328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.114424 4294 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded" Oct 31 21:47:03.159325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.158383 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 21:47:03.363513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.363404 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:47:03.529361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.529328 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz"] Oct 31 21:47:03.536342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.530290 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6a940e08-2935-495c-896a-74a979b821c6" containerName="payload-processing" Oct 31 21:47:03.536546 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.536528 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a940e08-2935-495c-896a-74a979b821c6" containerName="payload-processing" Oct 31 21:47:03.536713 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.536700 4294 cpu_manager.go:414] "containerMap: removing container" podUID="71ffeef8-93b1-4b63-922a-0f9b54b7b566" containerName="revoke-keys" Oct 31 21:47:03.536803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.536791 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ffeef8-93b1-4b63-922a-0f9b54b7b566" containerName="revoke-keys" Oct 31 21:47:03.536902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.536891 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" containerName="payload-pre-processing" Oct 31 21:47:03.536981 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.536969 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" containerName="payload-pre-processing" Oct 31 21:47:03.537241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.537225 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ffeef8-93b1-4b63-922a-0f9b54b7b566" containerName="revoke-keys" Oct 31 21:47:03.537432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.537418 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" containerName="payload-pre-processing" Oct 31 21:47:03.537529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.537518 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a940e08-2935-495c-896a-74a979b821c6" containerName="payload-processing" Oct 31 21:47:03.584326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.568074 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz"] Oct 31 21:47:03.584326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.568195 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.588115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.584824 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:03.588115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.586279 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:03.648628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.629580 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hwt\" (UniqueName: \"kubernetes.io/projected/e3d6b131-fe5f-4e1e-9563-fcad55dca802-kube-api-access-s7hwt\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.648628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.629764 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.648628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.629911 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.734244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.731093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hwt\" (UniqueName: \"kubernetes.io/projected/e3d6b131-fe5f-4e1e-9563-fcad55dca802-kube-api-access-s7hwt\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.734244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.731276 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.734244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.731431 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.761379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.756436 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.765760 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.765730 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.855463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.853126 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hwt\" (UniqueName: \"kubernetes.io/projected/e3d6b131-fe5f-4e1e-9563-fcad55dca802-kube-api-access-s7hwt\") pod \"maas-api-revoke-sub-d125d452-h2ffz\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:03.963216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:03.963122 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:04.003325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:04.000919 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a940e08-2935-495c-896a-74a979b821c6" path="/var/lib/kubelet/pods/6a940e08-2935-495c-896a-74a979b821c6/volumes" Oct 31 21:47:04.003325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:04.001583 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90c5062-d9a4-4f71-ae01-ea8f863ef1cd" path="/var/lib/kubelet/pods/e90c5062-d9a4-4f71-ae01-ea8f863ef1cd/volumes" Oct 31 21:47:04.618656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:04.618623 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh"] Oct 31 21:47:04.621957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:04.621929 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-1f523a-9kgmh"] Oct 31 21:47:04.643984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:04.643956 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz"] Oct 31 21:47:04.645013 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:04.644991 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d6b131_fe5f_4e1e_9563_fcad55dca802.slice/crio-b1f30c9a76123baf0916f2bcbe010ed78521cb0682b8c12129d6e23a0f795708 WatchSource:0}: Error finding container b1f30c9a76123baf0916f2bcbe010ed78521cb0682b8c12129d6e23a0f795708: Status 404 returned error can't find the container with id b1f30c9a76123baf0916f2bcbe010ed78521cb0682b8c12129d6e23a0f795708 Oct 31 21:47:04.938201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:04.938165 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" event={"ID":"e3d6b131-fe5f-4e1e-9563-fcad55dca802","Type":"ContainerStarted","Data":"b1f30c9a76123baf0916f2bcbe010ed78521cb0682b8c12129d6e23a0f795708"} Oct 31 21:47:05.947037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:05.947003 4294 generic.go:356] "Generic (PLEG): container finished" podID="e3d6b131-fe5f-4e1e-9563-fcad55dca802" containerID="c67bb9733a8a97e5454258009e1e71e9b29fbaa53dc98cef8a1bbfd3d9382c4e" exitCode=0 Oct 31 21:47:05.947473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:05.947095 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" event={"ID":"e3d6b131-fe5f-4e1e-9563-fcad55dca802","Type":"ContainerDied","Data":"c67bb9733a8a97e5454258009e1e71e9b29fbaa53dc98cef8a1bbfd3d9382c4e"} Oct 31 21:47:06.004394 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:06.004361 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf" path="/var/lib/kubelet/pods/0aff6a0b-fbf2-42a9-ba8c-84a255cecdbf/volumes" Oct 31 21:47:07.006170 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.005575 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl"] Oct 31 21:47:07.013563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.013542 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.033789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.033745 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl"] Oct 31 21:47:07.041338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041291 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43407457-00de-4bac-87a6-9842f359dbb5-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041378 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041435 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041499 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041531 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvjbd\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-kube-api-access-jvjbd\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041567 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041620 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041660 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43407457-00de-4bac-87a6-9842f359dbb5-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.041774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.041709 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.142761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.142724 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143040 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143084 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvjbd\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-kube-api-access-jvjbd\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143140 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143204 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143264 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43407457-00de-4bac-87a6-9842f359dbb5-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143342 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143446 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43407457-00de-4bac-87a6-9842f359dbb5-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143477 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143506 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143652 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-credential-socket\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143748 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-certs\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.143899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.143875 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-data\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.144479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.144458 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43407457-00de-4bac-87a6-9842f359dbb5-istiod-ca-cert\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.148328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.146196 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-envoy\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.148328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.146338 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43407457-00de-4bac-87a6-9842f359dbb5-istio-podinfo\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.167041 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.167020 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-istio-token\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.167263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.167241 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvjbd\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-kube-api-access-jvjbd\") pod \"e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.217106 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.217050 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:07.244682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.244656 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3d6b131-fe5f-4e1e-9563-fcad55dca802-kube-api-access-s7hwt\" (UniqueName: \"kubernetes.io/projected/e3d6b131-fe5f-4e1e-9563-fcad55dca802-kube-api-access-s7hwt\") pod \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " Oct 31 21:47:07.244818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.244723 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-xks-ca\") pod \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " Oct 31 21:47:07.244818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.244766 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-openshift-ca\") pod \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\" (UID: \"e3d6b131-fe5f-4e1e-9563-fcad55dca802\") " Oct 31 21:47:07.245134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.245107 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-openshift-ca" pod "e3d6b131-fe5f-4e1e-9563-fcad55dca802" (UID: "e3d6b131-fe5f-4e1e-9563-fcad55dca802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:07.245329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.245297 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:07.246755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.246725 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d6b131-fe5f-4e1e-9563-fcad55dca802-kube-api-access-s7hwt" pod "e3d6b131-fe5f-4e1e-9563-fcad55dca802" (UID: "e3d6b131-fe5f-4e1e-9563-fcad55dca802"). InnerVolumeSpecName "kube-api-access-s7hwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:07.246842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.246730 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-xks-ca" pod "e3d6b131-fe5f-4e1e-9563-fcad55dca802" (UID: "e3d6b131-fe5f-4e1e-9563-fcad55dca802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:07.339966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.339939 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:07.346042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.346012 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7hwt\" (UniqueName: \"kubernetes.io/projected/e3d6b131-fe5f-4e1e-9563-fcad55dca802-kube-api-access-s7hwt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:07.346042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.346042 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3d6b131-fe5f-4e1e-9563-fcad55dca802-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:07.705378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.705164 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl"] Oct 31 21:47:07.706344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.705961 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:07.706344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.706045 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:07.706344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.706091 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:07.962583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.962479 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" event={"ID":"43407457-00de-4bac-87a6-9842f359dbb5","Type":"ContainerStarted","Data":"d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac"} Oct 31 21:47:07.962583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.962544 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" event={"ID":"43407457-00de-4bac-87a6-9842f359dbb5","Type":"ContainerStarted","Data":"10fcbbce325acd3731c96761fe011c29116a623f67c89c03effcd8817579979c"} Oct 31 21:47:07.963981 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.963942 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" Oct 31 21:47:07.964110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.963993 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz" event={"ID":"e3d6b131-fe5f-4e1e-9563-fcad55dca802","Type":"ContainerDied","Data":"b1f30c9a76123baf0916f2bcbe010ed78521cb0682b8c12129d6e23a0f795708"} Oct 31 21:47:07.964110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:07.964054 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1f30c9a76123baf0916f2bcbe010ed78521cb0682b8c12129d6e23a0f795708" Oct 31 21:47:08.013671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:08.013613 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podStartSLOduration=2.013592888 podStartE2EDuration="2.013592888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:08.00806965 +0000 UTC m=+1868.133176872" watchObservedRunningTime="2026-10-31 21:47:08.013592888 +0000 UTC m=+1868.138700119" Oct 31 21:47:08.340413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:08.340383 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:08.342187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:08.342162 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 31 21:47:08.342337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:08.342217 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 31 21:47:09.341008 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:09.340973 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 31 21:47:09.341519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:09.341034 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 31 21:47:10.256361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.256319 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz"] Oct 31 21:47:10.256905 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.256890 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e3d6b131-fe5f-4e1e-9563-fcad55dca802" containerName="revoke-keys" Oct 31 21:47:10.256953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.256907 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3d6b131-fe5f-4e1e-9563-fcad55dca802" containerName="revoke-keys" Oct 31 21:47:10.257039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.257030 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3d6b131-fe5f-4e1e-9563-fcad55dca802" containerName="revoke-keys" Oct 31 21:47:10.300169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.299689 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.305092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.305060 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz"] Oct 31 21:47:10.314711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.314674 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:10.326231 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.326212 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:10.340887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.340862 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 31 21:47:10.340994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.340910 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 31 21:47:10.416692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.416660 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfk2\" (UniqueName: \"kubernetes.io/projected/17821818-4501-419d-898f-636cb0d56577-kube-api-access-5bfk2\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.417088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.416721 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821818-4501-419d-898f-636cb0d56577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.417088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.416743 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821818-4501-419d-898f-636cb0d56577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.518148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.518116 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfk2\" (UniqueName: \"kubernetes.io/projected/17821818-4501-419d-898f-636cb0d56577-kube-api-access-5bfk2\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.518335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.518180 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821818-4501-419d-898f-636cb0d56577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.518335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.518207 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821818-4501-419d-898f-636cb0d56577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.518910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.518877 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821818-4501-419d-898f-636cb0d56577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.520857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.520831 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821818-4501-419d-898f-636cb0d56577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.541498 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.541465 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfk2\" (UniqueName: \"kubernetes.io/projected/17821818-4501-419d-898f-636cb0d56577-kube-api-access-5bfk2\") pod \"maas-api-revoke-sub-0c05638a-28lgz\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.634841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.634810 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:10.665465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.665430 4294 patch_prober.go:29] interesting pod/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:47:10.665594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.665505 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:47:10.965274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:10.965244 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz"] Oct 31 21:47:10.966577 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:10.966550 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17821818_4501_419d_898f_636cb0d56577.slice/crio-ccfbdc4cc5f47406676cc6f82c9043351c677a4b0a6cf421fa6effc0fa661a73 WatchSource:0}: Error finding container ccfbdc4cc5f47406676cc6f82c9043351c677a4b0a6cf421fa6effc0fa661a73: Status 404 returned error can't find the container with id ccfbdc4cc5f47406676cc6f82c9043351c677a4b0a6cf421fa6effc0fa661a73 Oct 31 21:47:11.022734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:11.022701 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" event={"ID":"17821818-4501-419d-898f-636cb0d56577","Type":"ContainerStarted","Data":"ccfbdc4cc5f47406676cc6f82c9043351c677a4b0a6cf421fa6effc0fa661a73"} Oct 31 21:47:11.341365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:11.341320 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 31 21:47:11.341541 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:11.341388 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 31 21:47:12.031787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.031745 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" event={"ID":"17821818-4501-419d-898f-636cb0d56577","Type":"ContainerStarted","Data":"8ec36ca39094644d3624a8a43fbf0aa5c7ef8457dd969dbe6f5d88f97eeea5a3"} Oct 31 21:47:12.075281 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.075217 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" podStartSLOduration=2.075198016 podStartE2EDuration="2.075198016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:12.072089522 +0000 UTC m=+1872.197196772" watchObservedRunningTime="2026-10-31 21:47:12.075198016 +0000 UTC m=+1872.200305248" Oct 31 21:47:12.162347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.158396 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg"] Oct 31 21:47:12.162347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.158628 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" containerID="cri-o://2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939" gracePeriod=30 Oct 31 21:47:12.397383 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.361790 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 31 21:47:12.397383 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.361853 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 31 21:47:12.950815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.950724 4294 patch_prober.go:29] interesting pod/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 21:47:12.950815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:12.950784 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:47:13.040716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.040683 4294 generic.go:356] "Generic (PLEG): container finished" podID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerID="854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22" exitCode=0 Oct 31 21:47:13.041189 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.040756 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" event={"ID":"f4cc3f4e-9300-479c-b129-c0f12eed77f1","Type":"ContainerDied","Data":"854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22"} Oct 31 21:47:13.041251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.041217 4294 scope.go:122] "RemoveContainer" containerID="854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22" Oct 31 21:47:13.042658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.042637 4294 generic.go:356] "Generic (PLEG): container finished" podID="17821818-4501-419d-898f-636cb0d56577" containerID="8ec36ca39094644d3624a8a43fbf0aa5c7ef8457dd969dbe6f5d88f97eeea5a3" exitCode=0 Oct 31 21:47:13.042801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.042688 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" event={"ID":"17821818-4501-419d-898f-636cb0d56577","Type":"ContainerDied","Data":"8ec36ca39094644d3624a8a43fbf0aa5c7ef8457dd969dbe6f5d88f97eeea5a3"} Oct 31 21:47:13.042949 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.042924 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:13.043006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.042992 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:13.043041 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.043022 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:13.340603 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.340566 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 31 21:47:13.340779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:13.340622 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 31 21:47:14.055005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.054962 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" event={"ID":"f4cc3f4e-9300-479c-b129-c0f12eed77f1","Type":"ContainerStarted","Data":"8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa"} Oct 31 21:47:14.324632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.321893 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:14.341103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.341070 4294 patch_prober.go:29] interesting pod/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 31 21:47:14.341254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.341132 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 31 21:47:14.427128 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.427087 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17821818-4501-419d-898f-636cb0d56577-kube-api-access-5bfk2\" (UniqueName: \"kubernetes.io/projected/17821818-4501-419d-898f-636cb0d56577-kube-api-access-5bfk2\") pod \"17821818-4501-419d-898f-636cb0d56577\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " Oct 31 21:47:14.427499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.427474 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17821818-4501-419d-898f-636cb0d56577-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821818-4501-419d-898f-636cb0d56577-maas-api-openshift-ca\") pod \"17821818-4501-419d-898f-636cb0d56577\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " Oct 31 21:47:14.427875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.427852 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17821818-4501-419d-898f-636cb0d56577-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821818-4501-419d-898f-636cb0d56577-maas-api-xks-ca\") pod \"17821818-4501-419d-898f-636cb0d56577\" (UID: \"17821818-4501-419d-898f-636cb0d56577\") " Oct 31 21:47:14.429337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.429293 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17821818-4501-419d-898f-636cb0d56577-maas-api-openshift-ca" pod "17821818-4501-419d-898f-636cb0d56577" (UID: "17821818-4501-419d-898f-636cb0d56577"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:14.431341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.431277 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17821818-4501-419d-898f-636cb0d56577-maas-api-xks-ca" pod "17821818-4501-419d-898f-636cb0d56577" (UID: "17821818-4501-419d-898f-636cb0d56577"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:14.442364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.442334 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17821818-4501-419d-898f-636cb0d56577-kube-api-access-5bfk2" pod "17821818-4501-419d-898f-636cb0d56577" (UID: "17821818-4501-419d-898f-636cb0d56577"). InnerVolumeSpecName "kube-api-access-5bfk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:14.456480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.456455 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:47:14.457045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.456996 4294 patch_prober.go:29] interesting pod/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 21:47:14.457134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.457057 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 21:47:14.530710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.530663 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bfk2\" (UniqueName: \"kubernetes.io/projected/17821818-4501-419d-898f-636cb0d56577-kube-api-access-5bfk2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:14.530710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.530710 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17821818-4501-419d-898f-636cb0d56577-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:14.530972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:14.530727 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17821818-4501-419d-898f-636cb0d56577-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:15.067611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.067411 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" Oct 31 21:47:15.067611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.067526 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz" event={"ID":"17821818-4501-419d-898f-636cb0d56577","Type":"ContainerDied","Data":"ccfbdc4cc5f47406676cc6f82c9043351c677a4b0a6cf421fa6effc0fa661a73"} Oct 31 21:47:15.067611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.067565 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccfbdc4cc5f47406676cc6f82c9043351c677a4b0a6cf421fa6effc0fa661a73" Oct 31 21:47:15.456396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.456310 4294 patch_prober.go:29] interesting pod/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 21:47:15.456396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.456361 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 21:47:15.549192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.549155 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:15.549896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.549878 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:15.550928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:15.550912 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:47:16.455880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:16.455847 4294 patch_prober.go:29] interesting pod/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 21:47:16.456267 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:16.455901 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 21:47:17.221316 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.221270 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" podUID="29f0c6c5-d180-4725-a7e9-0727e74f4ecd" containerName="main" containerID="cri-o://2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75" gracePeriod=45 Oct 31 21:47:17.239088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.239055 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" podUID="29f0c6c5-d180-4725-a7e9-0727e74f4ecd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.109:8000/ready\": dial tcp 10.134.0.109:8000: connect: connection refused" Oct 31 21:47:17.457173 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.457099 4294 patch_prober.go:29] interesting pod/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 21:47:17.458285 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.457343 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 21:47:17.546336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.542509 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:47:17.615526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.615479 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:47:17.717953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.717868 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-envoy\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.717957 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-kube-api-access-dzhk8\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-kube-api-access-dzhk8\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.717996 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tls-certs\" (UniqueName: \"kubernetes.io/secret/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tls-certs\") pod \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " Oct 31 21:47:17.718131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718027 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-certs\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718053 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-home\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-home\") pod \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " Oct 31 21:47:17.718131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718098 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-istio-token\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-istio-token\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718129 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-dshm\") pod \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " Oct 31 21:47:17.718493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718159 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134fced6-b695-470e-b923-d59215873872-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134fced6-b695-470e-b923-d59215873872-istiod-ca-cert\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718221 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-credential-socket\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718345 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-socket\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718394 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/134fced6-b695-470e-b923-d59215873872-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134fced6-b695-470e-b923-d59215873872-istio-podinfo\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.718493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718484 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tmp-dir\") pod \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " Oct 31 21:47:17.718758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718518 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-kube-api-access-xh2wt\" (UniqueName: \"kubernetes.io/projected/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-kube-api-access-xh2wt\") pod \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " Oct 31 21:47:17.718758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718616 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-model-cache\") pod \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\" (UID: \"29f0c6c5-d180-4725-a7e9-0727e74f4ecd\") " Oct 31 21:47:17.718758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718619 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-home" pod "29f0c6c5-d180-4725-a7e9-0727e74f4ecd" (UID: "29f0c6c5-d180-4725-a7e9-0727e74f4ecd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.718758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.718644 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-data\") pod \"134fced6-b695-470e-b923-d59215873872\" (UID: \"134fced6-b695-470e-b923-d59215873872\") " Oct 31 21:47:17.719390 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.719367 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.720254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.720227 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-data" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.720391 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.720267 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-certs" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.721715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.721672 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-credential-socket" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.721904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.721880 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134fced6-b695-470e-b923-d59215873872-istiod-ca-cert" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:17.722365 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.722340 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-socket" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.723060 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.723012 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tmp-dir" pod "29f0c6c5-d180-4725-a7e9-0727e74f4ecd" (UID: "29f0c6c5-d180-4725-a7e9-0727e74f4ecd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.723252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.723230 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-model-cache" pod "29f0c6c5-d180-4725-a7e9-0727e74f4ecd" (UID: "29f0c6c5-d180-4725-a7e9-0727e74f4ecd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.727046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.727008 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-kube-api-access-xh2wt" pod "29f0c6c5-d180-4725-a7e9-0727e74f4ecd" (UID: "29f0c6c5-d180-4725-a7e9-0727e74f4ecd"). InnerVolumeSpecName "kube-api-access-xh2wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:17.727604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.727569 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-envoy" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.732759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.732650 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-kube-api-access-dzhk8" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "kube-api-access-dzhk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:17.732759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.732720 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tls-certs" pod "29f0c6c5-d180-4725-a7e9-0727e74f4ecd" (UID: "29f0c6c5-d180-4725-a7e9-0727e74f4ecd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:17.733131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.733025 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-dshm" pod "29f0c6c5-d180-4725-a7e9-0727e74f4ecd" (UID: "29f0c6c5-d180-4725-a7e9-0727e74f4ecd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:17.733409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.733389 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-istio-token" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:17.734475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.734458 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/134fced6-b695-470e-b923-d59215873872-istio-podinfo" pod "134fced6-b695-470e-b923-d59215873872" (UID: "134fced6-b695-470e-b923-d59215873872"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:47:17.821654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821616 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821656 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821670 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821684 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzhk8\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-kube-api-access-dzhk8\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821698 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821712 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821724 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134fced6-b695-470e-b923-d59215873872-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821736 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821749 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134fced6-b695-470e-b923-d59215873872-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821762 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821774 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134fced6-b695-470e-b923-d59215873872-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821789 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134fced6-b695-470e-b923-d59215873872-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821802 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:17.821853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:17.821818 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh2wt\" (UniqueName: \"kubernetes.io/projected/29f0c6c5-d180-4725-a7e9-0727e74f4ecd-kube-api-access-xh2wt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:18.048673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.048639 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4"] Oct 31 21:47:18.049156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049142 4294 cpu_manager.go:414] "containerMap: removing container" podUID="29f0c6c5-d180-4725-a7e9-0727e74f4ecd" containerName="main" Oct 31 21:47:18.049197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049157 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f0c6c5-d180-4725-a7e9-0727e74f4ecd" containerName="main" Oct 31 21:47:18.049197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049173 4294 cpu_manager.go:414] "containerMap: removing container" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" Oct 31 21:47:18.049197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049178 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" Oct 31 21:47:18.049197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049191 4294 cpu_manager.go:414] "containerMap: removing container" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" Oct 31 21:47:18.049197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049196 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" Oct 31 21:47:18.049359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049206 4294 cpu_manager.go:414] "containerMap: removing container" podUID="17821818-4501-419d-898f-636cb0d56577" containerName="revoke-keys" Oct 31 21:47:18.049359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049211 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17821818-4501-419d-898f-636cb0d56577" containerName="revoke-keys" Oct 31 21:47:18.049359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049286 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="17821818-4501-419d-898f-636cb0d56577" containerName="revoke-keys" Oct 31 21:47:18.049359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049317 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f0c6c5-d180-4725-a7e9-0727e74f4ecd" containerName="main" Oct 31 21:47:18.049359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049324 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" Oct 31 21:47:18.049359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.049332 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="134fced6-b695-470e-b923-d59215873872" containerName="istio-proxy" Oct 31 21:47:18.052260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.052239 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.067549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.067523 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:18.067665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.067590 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:18.096329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.072868 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4"] Oct 31 21:47:18.112045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.112013 4294 generic.go:356] "Generic (PLEG): container finished" podID="29f0c6c5-d180-4725-a7e9-0727e74f4ecd" containerID="2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75" exitCode=0 Oct 31 21:47:18.112225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.112205 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" event={"ID":"29f0c6c5-d180-4725-a7e9-0727e74f4ecd","Type":"ContainerDied","Data":"2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75"} Oct 31 21:47:18.112341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.112248 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" event={"ID":"29f0c6c5-d180-4725-a7e9-0727e74f4ecd","Type":"ContainerDied","Data":"86c47893fd2afd075f7ce11ee7681e6c5abbb03347bfc8178bfa191852c50b56"} Oct 31 21:47:18.112341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.112272 4294 scope.go:122] "RemoveContainer" containerID="2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75" Oct 31 21:47:18.112461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.112448 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5" Oct 31 21:47:18.116545 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.116518 4294 generic.go:356] "Generic (PLEG): container finished" podID="134fced6-b695-470e-b923-d59215873872" containerID="2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939" exitCode=0 Oct 31 21:47:18.116668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.116574 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" Oct 31 21:47:18.116668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.116585 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" event={"ID":"134fced6-b695-470e-b923-d59215873872","Type":"ContainerDied","Data":"2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939"} Oct 31 21:47:18.116668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.116634 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg" event={"ID":"134fced6-b695-470e-b923-d59215873872","Type":"ContainerDied","Data":"767e752f1cfc9608c4c09be32e1248b73b7f97ba5f66e693132da3cc67476de9"} Oct 31 21:47:18.125048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.125025 4294 scope.go:122] "RemoveContainer" containerID="2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75" Oct 31 21:47:18.125401 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:47:18.125375 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75\": container with ID starting with 2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75 not found: ID does not exist" containerID="2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75" Oct 31 21:47:18.125482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.125413 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75"} err="failed to get container status \"2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75\": rpc error: code = NotFound desc = could not find container \"2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75\": container with ID starting with 2d0ca8474406e6033b388c96cde794531a10278ad9e674047e334bd5e443ec75 not found: ID does not exist" Oct 31 21:47:18.125482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.125437 4294 scope.go:122] "RemoveContainer" containerID="2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939" Oct 31 21:47:18.135553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.135537 4294 scope.go:122] "RemoveContainer" containerID="52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb" Oct 31 21:47:18.145766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.145747 4294 scope.go:122] "RemoveContainer" containerID="2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939" Oct 31 21:47:18.146071 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:47:18.146047 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939\": container with ID starting with 2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939 not found: ID does not exist" containerID="2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939" Oct 31 21:47:18.146132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.146080 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939"} err="failed to get container status \"2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939\": rpc error: code = NotFound desc = could not find container \"2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939\": container with ID starting with 2e5c48a1b9abcc8b275600f2246f0748f76e833eb765fb6c57c3d472c6f2f939 not found: ID does not exist" Oct 31 21:47:18.146132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.146098 4294 scope.go:122] "RemoveContainer" containerID="52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb" Oct 31 21:47:18.146422 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:47:18.146396 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb\": container with ID starting with 52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb not found: ID does not exist" containerID="52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb" Oct 31 21:47:18.146496 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.146428 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb"} err="failed to get container status \"52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb\": rpc error: code = NotFound desc = could not find container \"52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb\": container with ID starting with 52cd0dd9374ca64e989bccf6fe1ebf153068be0691122b63c2ce466283360dbb not found: ID does not exist" Oct 31 21:47:18.165697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.165672 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5"] Oct 31 21:47:18.200523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.200494 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-80542b/praxis-route-80542b-kserve-78857d84b7-mmrg5"] Oct 31 21:47:18.212736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.212711 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg"] Oct 31 21:47:18.217011 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.216986 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-1f523a-openshift-default-6bf49b5765-b9qcg"] Oct 31 21:47:18.226999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.226973 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.227101 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.227027 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ts7z\" (UniqueName: \"kubernetes.io/projected/10e472a3-3c0f-4567-b56d-4a74047ceb02-kube-api-access-2ts7z\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.227579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.227148 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.328347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.328212 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.328347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.328285 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ts7z\" (UniqueName: \"kubernetes.io/projected/10e472a3-3c0f-4567-b56d-4a74047ceb02-kube-api-access-2ts7z\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.328593 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.328394 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.329094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.329067 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.330544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.330524 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.351205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.351180 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ts7z\" (UniqueName: \"kubernetes.io/projected/10e472a3-3c0f-4567-b56d-4a74047ceb02-kube-api-access-2ts7z\") pod \"maas-api-revoke-sub-40434452-42bb4\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.364218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.364195 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:18.460507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.460042 4294 patch_prober.go:29] interesting pod/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 31 21:47:18.460507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.460096 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 31 21:47:18.723032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:18.723007 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4"] Oct 31 21:47:18.724155 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:18.724126 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10e472a3_3c0f_4567_b56d_4a74047ceb02.slice/crio-1e0c3ee7969636f93c9742f0976599a6153b868c2077ce4900d360e9fab8870f WatchSource:0}: Error finding container 1e0c3ee7969636f93c9742f0976599a6153b868c2077ce4900d360e9fab8870f: Status 404 returned error can't find the container with id 1e0c3ee7969636f93c9742f0976599a6153b868c2077ce4900d360e9fab8870f Oct 31 21:47:19.126579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.126550 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" event={"ID":"10e472a3-3c0f-4567-b56d-4a74047ceb02","Type":"ContainerStarted","Data":"fd3d9e696a16096c4c5f39858863a2542d73adf66733509805b5fb5795efdba0"} Oct 31 21:47:19.126690 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.126601 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" event={"ID":"10e472a3-3c0f-4567-b56d-4a74047ceb02","Type":"ContainerStarted","Data":"1e0c3ee7969636f93c9742f0976599a6153b868c2077ce4900d360e9fab8870f"} Oct 31 21:47:19.167274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.167221 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" podStartSLOduration=1.167204728 podStartE2EDuration="1.167204728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:19.166006709 +0000 UTC m=+1879.291113951" watchObservedRunningTime="2026-10-31 21:47:19.167204728 +0000 UTC m=+1879.292311960" Oct 31 21:47:19.176618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.176588 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx"] Oct 31 21:47:19.179907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.179889 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.195997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.195970 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ad121b\"" Oct 31 21:47:19.196775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.196753 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ad121b-dockercfg-px9k5\"" Oct 31 21:47:19.205276 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.205250 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx"] Oct 31 21:47:19.240199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.240160 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.240492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.240468 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvx4x\" (UniqueName: \"kubernetes.io/projected/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-kube-api-access-tvx4x\") pod \"payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.348322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.347351 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.348322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.347606 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvx4x\" (UniqueName: \"kubernetes.io/projected/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-kube-api-access-tvx4x\") pod \"payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.359637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.356544 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.408322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.407438 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvx4x\" (UniqueName: \"kubernetes.io/projected/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-kube-api-access-tvx4x\") pod \"payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.495283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.495245 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:19.498393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.498357 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:47:19.498812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.498793 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:47:19.499775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.499753 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:47:19.824294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.824262 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx"] Oct 31 21:47:19.985170 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.985136 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134fced6-b695-470e-b923-d59215873872" path="/var/lib/kubelet/pods/134fced6-b695-470e-b923-d59215873872/volumes" Oct 31 21:47:19.985627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:19.985612 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f0c6c5-d180-4725-a7e9-0727e74f4ecd" path="/var/lib/kubelet/pods/29f0c6c5-d180-4725-a7e9-0727e74f4ecd/volumes" Oct 31 21:47:20.134941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:20.134858 4294 generic.go:356] "Generic (PLEG): container finished" podID="10e472a3-3c0f-4567-b56d-4a74047ceb02" containerID="fd3d9e696a16096c4c5f39858863a2542d73adf66733509805b5fb5795efdba0" exitCode=0 Oct 31 21:47:20.135076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:20.134942 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" event={"ID":"10e472a3-3c0f-4567-b56d-4a74047ceb02","Type":"ContainerDied","Data":"fd3d9e696a16096c4c5f39858863a2542d73adf66733509805b5fb5795efdba0"} Oct 31 21:47:20.136686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:20.136659 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" event={"ID":"34b3dab4-e9e1-42c6-baa9-1ad202b611eb","Type":"ContainerStarted","Data":"d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87"} Oct 31 21:47:20.136819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:20.136704 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" event={"ID":"34b3dab4-e9e1-42c6-baa9-1ad202b611eb","Type":"ContainerStarted","Data":"744b39c43c3489374d6f45941c793594b9e44daedca292d3e11a8c61043b8fa0"} Oct 31 21:47:20.136900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:20.136884 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:20.217402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:20.217333 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" podStartSLOduration=1.217318939 podStartE2EDuration="1.217318939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:20.214342831 +0000 UTC m=+1880.339450063" watchObservedRunningTime="2026-10-31 21:47:20.217318939 +0000 UTC m=+1880.342426164" Oct 31 21:47:21.328268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.328241 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:21.379992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.379957 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10e472a3-3c0f-4567-b56d-4a74047ceb02-kube-api-access-2ts7z\" (UniqueName: \"kubernetes.io/projected/10e472a3-3c0f-4567-b56d-4a74047ceb02-kube-api-access-2ts7z\") pod \"10e472a3-3c0f-4567-b56d-4a74047ceb02\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " Oct 31 21:47:21.380169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.380103 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-openshift-ca\") pod \"10e472a3-3c0f-4567-b56d-4a74047ceb02\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " Oct 31 21:47:21.380269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.380250 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-xks-ca\") pod \"10e472a3-3c0f-4567-b56d-4a74047ceb02\" (UID: \"10e472a3-3c0f-4567-b56d-4a74047ceb02\") " Oct 31 21:47:21.380551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.380517 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-openshift-ca" pod "10e472a3-3c0f-4567-b56d-4a74047ceb02" (UID: "10e472a3-3c0f-4567-b56d-4a74047ceb02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:21.381241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.381216 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:21.382571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.382537 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-xks-ca" pod "10e472a3-3c0f-4567-b56d-4a74047ceb02" (UID: "10e472a3-3c0f-4567-b56d-4a74047ceb02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:21.382730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.382705 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10e472a3-3c0f-4567-b56d-4a74047ceb02-kube-api-access-2ts7z" pod "10e472a3-3c0f-4567-b56d-4a74047ceb02" (UID: "10e472a3-3c0f-4567-b56d-4a74047ceb02"). InnerVolumeSpecName "kube-api-access-2ts7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:21.482747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.482720 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ts7z\" (UniqueName: \"kubernetes.io/projected/10e472a3-3c0f-4567-b56d-4a74047ceb02-kube-api-access-2ts7z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:21.482747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:21.482750 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10e472a3-3c0f-4567-b56d-4a74047ceb02-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:22.152741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.152707 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" Oct 31 21:47:22.152911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.152747 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4" event={"ID":"10e472a3-3c0f-4567-b56d-4a74047ceb02","Type":"ContainerDied","Data":"1e0c3ee7969636f93c9742f0976599a6153b868c2077ce4900d360e9fab8870f"} Oct 31 21:47:22.152911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.152804 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e0c3ee7969636f93c9742f0976599a6153b868c2077ce4900d360e9fab8870f" Oct 31 21:47:22.521523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.521492 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8"] Oct 31 21:47:22.522056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.522043 4294 cpu_manager.go:414] "containerMap: removing container" podUID="10e472a3-3c0f-4567-b56d-4a74047ceb02" containerName="revoke-keys" Oct 31 21:47:22.522108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.522057 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10e472a3-3c0f-4567-b56d-4a74047ceb02" containerName="revoke-keys" Oct 31 21:47:22.522139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.522132 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="10e472a3-3c0f-4567-b56d-4a74047ceb02" containerName="revoke-keys" Oct 31 21:47:22.524617 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.524601 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.550968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.550941 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:22.551122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.551019 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:22.555137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.555107 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8"] Oct 31 21:47:22.594876 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.594846 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4n9s\" (UniqueName: \"kubernetes.io/projected/936fa442-323f-4f8a-9c10-523bd7ec6475-kube-api-access-l4n9s\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.595118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.595090 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.595211 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.595192 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.696801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.696766 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.696970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.696847 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.696970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.696887 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4n9s\" (UniqueName: \"kubernetes.io/projected/936fa442-323f-4f8a-9c10-523bd7ec6475-kube-api-access-l4n9s\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.697530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.697505 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.699184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.699159 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.720052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.720025 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4n9s\" (UniqueName: \"kubernetes.io/projected/936fa442-323f-4f8a-9c10-523bd7ec6475-kube-api-access-l4n9s\") pod \"maas-api-revoke-sub-b54a35fc-s4dv8\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:22.836361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:22.836261 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:23.147413 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:23.147379 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936fa442_323f_4f8a_9c10_523bd7ec6475.slice/crio-33089129f2d3c259735ce6e4b774116cad0dd5e639c714c96b54576535d71439 WatchSource:0}: Error finding container 33089129f2d3c259735ce6e4b774116cad0dd5e639c714c96b54576535d71439: Status 404 returned error can't find the container with id 33089129f2d3c259735ce6e4b774116cad0dd5e639c714c96b54576535d71439 Oct 31 21:47:23.148632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:23.148605 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8"] Oct 31 21:47:23.175243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:23.175202 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" event={"ID":"936fa442-323f-4f8a-9c10-523bd7ec6475","Type":"ContainerStarted","Data":"33089129f2d3c259735ce6e4b774116cad0dd5e639c714c96b54576535d71439"} Oct 31 21:47:23.884031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:23.883998 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs"] Oct 31 21:47:23.884472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:23.884230 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" containerID="cri-o://b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b" gracePeriod=30 Oct 31 21:47:24.203741 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:24.202660 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" event={"ID":"936fa442-323f-4f8a-9c10-523bd7ec6475","Type":"ContainerStarted","Data":"937726b9f29573157dc89c3dda9d6ca158e130802896a20c99cc955b73f45731"} Oct 31 21:47:24.260159 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:24.260107 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" podStartSLOduration=2.260091264 podStartE2EDuration="2.260091264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:24.250042843 +0000 UTC m=+1884.375150073" watchObservedRunningTime="2026-10-31 21:47:24.260091264 +0000 UTC m=+1884.385198494" Oct 31 21:47:25.247396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:25.242343 4294 generic.go:356] "Generic (PLEG): container finished" podID="936fa442-323f-4f8a-9c10-523bd7ec6475" containerID="937726b9f29573157dc89c3dda9d6ca158e130802896a20c99cc955b73f45731" exitCode=0 Oct 31 21:47:25.247396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:25.242502 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" event={"ID":"936fa442-323f-4f8a-9c10-523bd7ec6475","Type":"ContainerDied","Data":"937726b9f29573157dc89c3dda9d6ca158e130802896a20c99cc955b73f45731"} Oct 31 21:47:25.379326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:25.378473 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-gmgjs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:47:25.379326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:25.378536 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:47:26.417980 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.417954 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:26.538627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.538533 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936fa442-323f-4f8a-9c10-523bd7ec6475-kube-api-access-l4n9s\" (UniqueName: \"kubernetes.io/projected/936fa442-323f-4f8a-9c10-523bd7ec6475-kube-api-access-l4n9s\") pod \"936fa442-323f-4f8a-9c10-523bd7ec6475\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " Oct 31 21:47:26.538806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.538689 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-xks-ca\") pod \"936fa442-323f-4f8a-9c10-523bd7ec6475\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " Oct 31 21:47:26.538870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.538817 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-openshift-ca\") pod \"936fa442-323f-4f8a-9c10-523bd7ec6475\" (UID: \"936fa442-323f-4f8a-9c10-523bd7ec6475\") " Oct 31 21:47:26.539286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.539239 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-openshift-ca" pod "936fa442-323f-4f8a-9c10-523bd7ec6475" (UID: "936fa442-323f-4f8a-9c10-523bd7ec6475"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:26.539534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.539513 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:26.540871 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.540839 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936fa442-323f-4f8a-9c10-523bd7ec6475-kube-api-access-l4n9s" pod "936fa442-323f-4f8a-9c10-523bd7ec6475" (UID: "936fa442-323f-4f8a-9c10-523bd7ec6475"). InnerVolumeSpecName "kube-api-access-l4n9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:26.541221 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.541186 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-xks-ca" pod "936fa442-323f-4f8a-9c10-523bd7ec6475" (UID: "936fa442-323f-4f8a-9c10-523bd7ec6475"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:26.640795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.640759 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4n9s\" (UniqueName: \"kubernetes.io/projected/936fa442-323f-4f8a-9c10-523bd7ec6475-kube-api-access-l4n9s\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:26.640795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:26.640788 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/936fa442-323f-4f8a-9c10-523bd7ec6475-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:27.258812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:27.258785 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" Oct 31 21:47:27.258959 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:27.258825 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8" event={"ID":"936fa442-323f-4f8a-9c10-523bd7ec6475","Type":"ContainerDied","Data":"33089129f2d3c259735ce6e4b774116cad0dd5e639c714c96b54576535d71439"} Oct 31 21:47:27.258959 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:27.258873 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33089129f2d3c259735ce6e4b774116cad0dd5e639c714c96b54576535d71439" Oct 31 21:47:29.158632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.158566 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:47:29.271030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271003 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-envoy\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271037 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-data\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271151 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-certs\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271197 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273e8c40-6b3c-4c53-8476-e527511c7340-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e8c40-6b3c-4c53-8476-e527511c7340-istiod-ca-cert\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271249 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-kube-api-access-dlsmc\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-kube-api-access-dlsmc\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271429 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-istio-token\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-istio-token\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271410 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-data" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:29.271512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271475 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-socket\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271498 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-certs" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:29.271623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271519 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/273e8c40-6b3c-4c53-8476-e527511c7340-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e8c40-6b3c-4c53-8476-e527511c7340-istio-podinfo\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271644 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-credential-socket\") pod \"273e8c40-6b3c-4c53-8476-e527511c7340\" (UID: \"273e8c40-6b3c-4c53-8476-e527511c7340\") " Oct 31 21:47:29.271760 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271734 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-socket" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:29.271810 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.271760 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273e8c40-6b3c-4c53-8476-e527511c7340-istiod-ca-cert" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:29.272032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.272003 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-credential-socket" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:29.272347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.272290 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.272460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.272352 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.272460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.272366 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.272460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.272377 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273e8c40-6b3c-4c53-8476-e527511c7340-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.272460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.272389 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.273632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.273500 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-envoy" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:47:29.273632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.273603 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-kube-api-access-dlsmc" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "kube-api-access-dlsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:29.273783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.273759 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-istio-token" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:29.273925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.273902 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/273e8c40-6b3c-4c53-8476-e527511c7340-istio-podinfo" pod "273e8c40-6b3c-4c53-8476-e527511c7340" (UID: "273e8c40-6b3c-4c53-8476-e527511c7340"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:47:29.275654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.275634 4294 generic.go:356] "Generic (PLEG): container finished" podID="273e8c40-6b3c-4c53-8476-e527511c7340" containerID="b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b" exitCode=0 Oct 31 21:47:29.275725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.275686 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" Oct 31 21:47:29.275761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.275720 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" event={"ID":"273e8c40-6b3c-4c53-8476-e527511c7340","Type":"ContainerDied","Data":"b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b"} Oct 31 21:47:29.275793 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.275775 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs" event={"ID":"273e8c40-6b3c-4c53-8476-e527511c7340","Type":"ContainerDied","Data":"14bddf84790876d062b750253277dfae4a33a97855f85dbf44f41e6092a17c2d"} Oct 31 21:47:29.275833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.275806 4294 scope.go:122] "RemoveContainer" containerID="b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b" Oct 31 21:47:29.301980 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.301962 4294 scope.go:122] "RemoveContainer" containerID="af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741" Oct 31 21:47:29.310850 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.310834 4294 scope.go:122] "RemoveContainer" containerID="b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b" Oct 31 21:47:29.311110 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:47:29.311093 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b\": container with ID starting with b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b not found: ID does not exist" containerID="b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b" Oct 31 21:47:29.311165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.311127 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b"} err="failed to get container status \"b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b\": rpc error: code = NotFound desc = could not find container \"b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b\": container with ID starting with b6a26f0e3e71aa0c0ffa39689583c9634ebe63b99c7c558571d7e04ca0d5988b not found: ID does not exist" Oct 31 21:47:29.311165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.311147 4294 scope.go:122] "RemoveContainer" containerID="af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741" Oct 31 21:47:29.311444 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:47:29.311425 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741\": container with ID starting with af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741 not found: ID does not exist" containerID="af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741" Oct 31 21:47:29.311497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.311450 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741"} err="failed to get container status \"af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741\": rpc error: code = NotFound desc = could not find container \"af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741\": container with ID starting with af3da13b03dcd891e5f72ae964295195df9c2ee0bb52bbec427444d2fc7c5741 not found: ID does not exist" Oct 31 21:47:29.337519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.337488 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs"] Oct 31 21:47:29.340878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.340857 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-gmgjs"] Oct 31 21:47:29.373483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.373456 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273e8c40-6b3c-4c53-8476-e527511c7340-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.373483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.373481 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlsmc\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-kube-api-access-dlsmc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.373642 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.373493 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273e8c40-6b3c-4c53-8476-e527511c7340-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.373642 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.373503 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273e8c40-6b3c-4c53-8476-e527511c7340-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:29.985430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:29.985396 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" path="/var/lib/kubelet/pods/273e8c40-6b3c-4c53-8476-e527511c7340/volumes" Oct 31 21:47:30.218386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.218351 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl"] Oct 31 21:47:30.218880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.218866 4294 cpu_manager.go:414] "containerMap: removing container" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" Oct 31 21:47:30.218924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.218880 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" Oct 31 21:47:30.218924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.218895 4294 cpu_manager.go:414] "containerMap: removing container" podUID="936fa442-323f-4f8a-9c10-523bd7ec6475" containerName="revoke-keys" Oct 31 21:47:30.218924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.218900 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936fa442-323f-4f8a-9c10-523bd7ec6475" containerName="revoke-keys" Oct 31 21:47:30.218924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.218912 4294 cpu_manager.go:414] "containerMap: removing container" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" Oct 31 21:47:30.218924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.218917 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" Oct 31 21:47:30.219066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.219010 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" Oct 31 21:47:30.219066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.219018 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="936fa442-323f-4f8a-9c10-523bd7ec6475" containerName="revoke-keys" Oct 31 21:47:30.219066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.219024 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="273e8c40-6b3c-4c53-8476-e527511c7340" containerName="istio-proxy" Oct 31 21:47:30.223759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.223741 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.241363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.241284 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl"] Oct 31 21:47:30.248210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.248178 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:30.248297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.248267 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:30.284726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.284696 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.284881 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.284807 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pnf\" (UniqueName: \"kubernetes.io/projected/36701014-e0e5-4f90-a53f-4de09038bbe3-kube-api-access-66pnf\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.284961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.284936 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.385869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.385835 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.386061 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.385917 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pnf\" (UniqueName: \"kubernetes.io/projected/36701014-e0e5-4f90-a53f-4de09038bbe3-kube-api-access-66pnf\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.386061 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.386037 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.386688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.386656 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.388421 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.388399 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.409147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.409123 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pnf\" (UniqueName: \"kubernetes.io/projected/36701014-e0e5-4f90-a53f-4de09038bbe3-kube-api-access-66pnf\") pod \"maas-api-revoke-sub-b5790797-sknwl\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.537769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.537676 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:30.844782 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:30.844753 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36701014_e0e5_4f90_a53f_4de09038bbe3.slice/crio-e1f3fcb4be731a2405bf66b015674ddbb530ef295626e957309abd8915406fe5 WatchSource:0}: Error finding container e1f3fcb4be731a2405bf66b015674ddbb530ef295626e957309abd8915406fe5: Status 404 returned error can't find the container with id e1f3fcb4be731a2405bf66b015674ddbb530ef295626e957309abd8915406fe5 Oct 31 21:47:30.845199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:30.845176 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl"] Oct 31 21:47:31.147815 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:31.147726 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:47:31.295031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:31.294991 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" event={"ID":"36701014-e0e5-4f90-a53f-4de09038bbe3","Type":"ContainerStarted","Data":"bc0a6ce1e17c3c079a4feb5700873efcc94d7adc4699d757c91dc0d08c52e069"} Oct 31 21:47:31.295462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:31.295058 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" event={"ID":"36701014-e0e5-4f90-a53f-4de09038bbe3","Type":"ContainerStarted","Data":"e1f3fcb4be731a2405bf66b015674ddbb530ef295626e957309abd8915406fe5"} Oct 31 21:47:31.316180 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:47:31.316143 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:47:31.336408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:31.336361 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" podStartSLOduration=1.336346933 podStartE2EDuration="1.336346933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:31.334581519 +0000 UTC m=+1891.459688751" watchObservedRunningTime="2026-10-31 21:47:31.336346933 +0000 UTC m=+1891.461454182" Oct 31 21:47:32.303604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:32.303569 4294 generic.go:356] "Generic (PLEG): container finished" podID="36701014-e0e5-4f90-a53f-4de09038bbe3" containerID="bc0a6ce1e17c3c079a4feb5700873efcc94d7adc4699d757c91dc0d08c52e069" exitCode=0 Oct 31 21:47:32.304108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:32.303661 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" event={"ID":"36701014-e0e5-4f90-a53f-4de09038bbe3","Type":"ContainerDied","Data":"bc0a6ce1e17c3c079a4feb5700873efcc94d7adc4699d757c91dc0d08c52e069"} Oct 31 21:47:33.477463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.477433 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:33.632087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.632014 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-xks-ca\") pod \"36701014-e0e5-4f90-a53f-4de09038bbe3\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " Oct 31 21:47:33.632241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.632230 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-openshift-ca\") pod \"36701014-e0e5-4f90-a53f-4de09038bbe3\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " Oct 31 21:47:33.632281 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.632262 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36701014-e0e5-4f90-a53f-4de09038bbe3-kube-api-access-66pnf\" (UniqueName: \"kubernetes.io/projected/36701014-e0e5-4f90-a53f-4de09038bbe3-kube-api-access-66pnf\") pod \"36701014-e0e5-4f90-a53f-4de09038bbe3\" (UID: \"36701014-e0e5-4f90-a53f-4de09038bbe3\") " Oct 31 21:47:33.632592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.632566 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-openshift-ca" pod "36701014-e0e5-4f90-a53f-4de09038bbe3" (UID: "36701014-e0e5-4f90-a53f-4de09038bbe3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:33.632751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.632720 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:33.634142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.634114 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-xks-ca" pod "36701014-e0e5-4f90-a53f-4de09038bbe3" (UID: "36701014-e0e5-4f90-a53f-4de09038bbe3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:33.634353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.634335 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36701014-e0e5-4f90-a53f-4de09038bbe3-kube-api-access-66pnf" pod "36701014-e0e5-4f90-a53f-4de09038bbe3" (UID: "36701014-e0e5-4f90-a53f-4de09038bbe3"). InnerVolumeSpecName "kube-api-access-66pnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:33.733604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.733575 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pnf\" (UniqueName: \"kubernetes.io/projected/36701014-e0e5-4f90-a53f-4de09038bbe3-kube-api-access-66pnf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:33.733604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:33.733600 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36701014-e0e5-4f90-a53f-4de09038bbe3-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:34.320392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:34.320290 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" Oct 31 21:47:34.320526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:34.320285 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl" event={"ID":"36701014-e0e5-4f90-a53f-4de09038bbe3","Type":"ContainerDied","Data":"e1f3fcb4be731a2405bf66b015674ddbb530ef295626e957309abd8915406fe5"} Oct 31 21:47:34.320526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:34.320440 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1f3fcb4be731a2405bf66b015674ddbb530ef295626e957309abd8915406fe5" Oct 31 21:47:37.114918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.114886 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q"] Oct 31 21:47:37.115497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.115482 4294 cpu_manager.go:414] "containerMap: removing container" podUID="36701014-e0e5-4f90-a53f-4de09038bbe3" containerName="revoke-keys" Oct 31 21:47:37.115540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.115498 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36701014-e0e5-4f90-a53f-4de09038bbe3" containerName="revoke-keys" Oct 31 21:47:37.115607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.115598 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="36701014-e0e5-4f90-a53f-4de09038bbe3" containerName="revoke-keys" Oct 31 21:47:37.119047 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.119032 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.138669 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.138636 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q"] Oct 31 21:47:37.145194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.145148 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:37.145315 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.145215 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:37.191827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.191799 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh"] Oct 31 21:47:37.195936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.195916 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.197648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.197622 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.197799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.197770 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.197894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.197872 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxgnc\" (UniqueName: \"kubernetes.io/projected/62ce57a0-d866-41c4-840a-cad25c4c46e0-kube-api-access-dxgnc\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.217038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.217001 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh"] Oct 31 21:47:37.299250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.299209 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.299461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.299392 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.299461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.299426 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.299540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.299467 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.299574 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.299552 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxgnc\" (UniqueName: \"kubernetes.io/projected/62ce57a0-d866-41c4-840a-cad25c4c46e0-kube-api-access-dxgnc\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.299615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.299603 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf5m2\" (UniqueName: \"kubernetes.io/projected/fb753653-a7e5-46cc-aaf8-d2ce11439a00-kube-api-access-vf5m2\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.300108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.300089 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.301787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.301768 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.323176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.323146 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxgnc\" (UniqueName: \"kubernetes.io/projected/62ce57a0-d866-41c4-840a-cad25c4c46e0-kube-api-access-dxgnc\") pod \"maas-api-revoke-sub-73d3dd93-bxg5q\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.400856 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.400775 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf5m2\" (UniqueName: \"kubernetes.io/projected/fb753653-a7e5-46cc-aaf8-d2ce11439a00-kube-api-access-vf5m2\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.400856 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.400843 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.401044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.400960 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.401619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.401601 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.403122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.403099 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.424220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.424192 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf5m2\" (UniqueName: \"kubernetes.io/projected/fb753653-a7e5-46cc-aaf8-d2ce11439a00-kube-api-access-vf5m2\") pod \"maas-api-revoke-sub-5e8ba20b-tlkgh\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.432795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.432775 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:37.510057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.510030 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:37.739396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.739365 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q"] Oct 31 21:47:37.739981 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:37.739953 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ce57a0_d866_41c4_840a_cad25c4c46e0.slice/crio-799332a7cbcbbd1e213b3df68884aa8b30bc9969d47ea696670482ff2d7e4a5f WatchSource:0}: Error finding container 799332a7cbcbbd1e213b3df68884aa8b30bc9969d47ea696670482ff2d7e4a5f: Status 404 returned error can't find the container with id 799332a7cbcbbd1e213b3df68884aa8b30bc9969d47ea696670482ff2d7e4a5f Oct 31 21:47:37.851828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:37.851779 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh"] Oct 31 21:47:37.867989 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:37.867132 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb753653_a7e5_46cc_aaf8_d2ce11439a00.slice/crio-5041ad6b5d63edd65ebf846d675c8107c6470cfd4f05de552307487ff481b357 WatchSource:0}: Error finding container 5041ad6b5d63edd65ebf846d675c8107c6470cfd4f05de552307487ff481b357: Status 404 returned error can't find the container with id 5041ad6b5d63edd65ebf846d675c8107c6470cfd4f05de552307487ff481b357 Oct 31 21:47:38.349375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:38.349332 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" event={"ID":"fb753653-a7e5-46cc-aaf8-d2ce11439a00","Type":"ContainerStarted","Data":"44bd7714323e7f191364c98a14622692def177c139c7a558c9b403ed5854c9e9"} Oct 31 21:47:38.349781 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:38.349395 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" event={"ID":"fb753653-a7e5-46cc-aaf8-d2ce11439a00","Type":"ContainerStarted","Data":"5041ad6b5d63edd65ebf846d675c8107c6470cfd4f05de552307487ff481b357"} Oct 31 21:47:38.351010 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:38.350986 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" event={"ID":"62ce57a0-d866-41c4-840a-cad25c4c46e0","Type":"ContainerStarted","Data":"27dd9206599f6bb1530d3fe0a7925455cb5c9d8b12ebe8eb5a0c7af5b655369c"} Oct 31 21:47:38.351135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:38.351034 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" event={"ID":"62ce57a0-d866-41c4-840a-cad25c4c46e0","Type":"ContainerStarted","Data":"799332a7cbcbbd1e213b3df68884aa8b30bc9969d47ea696670482ff2d7e4a5f"} Oct 31 21:47:38.391984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:38.391942 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" podStartSLOduration=1.391930101 podStartE2EDuration="1.391930101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:38.389653497 +0000 UTC m=+1898.514760728" watchObservedRunningTime="2026-10-31 21:47:38.391930101 +0000 UTC m=+1898.517037331" Oct 31 21:47:38.433014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:38.432963 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" podStartSLOduration=1.432948715 podStartE2EDuration="1.432948715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:38.430724944 +0000 UTC m=+1898.555832174" watchObservedRunningTime="2026-10-31 21:47:38.432948715 +0000 UTC m=+1898.558055964" Oct 31 21:47:39.511683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:39.511646 4294 generic.go:356] "Generic (PLEG): container finished" podID="62ce57a0-d866-41c4-840a-cad25c4c46e0" containerID="27dd9206599f6bb1530d3fe0a7925455cb5c9d8b12ebe8eb5a0c7af5b655369c" exitCode=0 Oct 31 21:47:39.512386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:39.512358 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" event={"ID":"62ce57a0-d866-41c4-840a-cad25c4c46e0","Type":"ContainerDied","Data":"27dd9206599f6bb1530d3fe0a7925455cb5c9d8b12ebe8eb5a0c7af5b655369c"} Oct 31 21:47:39.528114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:39.528085 4294 generic.go:356] "Generic (PLEG): container finished" podID="fb753653-a7e5-46cc-aaf8-d2ce11439a00" containerID="44bd7714323e7f191364c98a14622692def177c139c7a558c9b403ed5854c9e9" exitCode=0 Oct 31 21:47:39.528457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:39.528432 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" event={"ID":"fb753653-a7e5-46cc-aaf8-d2ce11439a00","Type":"ContainerDied","Data":"44bd7714323e7f191364c98a14622692def177c139c7a558c9b403ed5854c9e9"} Oct 31 21:47:40.924607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:40.924576 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:40.943801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:40.943778 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:41.092373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.092272 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-xks-ca\") pod \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " Oct 31 21:47:41.092556 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.092534 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-openshift-ca\") pod \"62ce57a0-d866-41c4-840a-cad25c4c46e0\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " Oct 31 21:47:41.092619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.092574 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ce57a0-d866-41c4-840a-cad25c4c46e0-kube-api-access-dxgnc\" (UniqueName: \"kubernetes.io/projected/62ce57a0-d866-41c4-840a-cad25c4c46e0-kube-api-access-dxgnc\") pod \"62ce57a0-d866-41c4-840a-cad25c4c46e0\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " Oct 31 21:47:41.092660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.092642 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-openshift-ca\") pod \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " Oct 31 21:47:41.092699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.092675 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-xks-ca\") pod \"62ce57a0-d866-41c4-840a-cad25c4c46e0\" (UID: \"62ce57a0-d866-41c4-840a-cad25c4c46e0\") " Oct 31 21:47:41.092740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.092699 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb753653-a7e5-46cc-aaf8-d2ce11439a00-kube-api-access-vf5m2\" (UniqueName: \"kubernetes.io/projected/fb753653-a7e5-46cc-aaf8-d2ce11439a00-kube-api-access-vf5m2\") pod \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\" (UID: \"fb753653-a7e5-46cc-aaf8-d2ce11439a00\") " Oct 31 21:47:41.093007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.092983 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-openshift-ca" pod "62ce57a0-d866-41c4-840a-cad25c4c46e0" (UID: "62ce57a0-d866-41c4-840a-cad25c4c46e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:41.093108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.093060 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-openshift-ca" pod "fb753653-a7e5-46cc-aaf8-d2ce11439a00" (UID: "fb753653-a7e5-46cc-aaf8-d2ce11439a00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:41.093495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.093471 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:41.093592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.093498 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:41.094960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.094932 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-xks-ca" pod "62ce57a0-d866-41c4-840a-cad25c4c46e0" (UID: "62ce57a0-d866-41c4-840a-cad25c4c46e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:41.095058 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.095010 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-xks-ca" pod "fb753653-a7e5-46cc-aaf8-d2ce11439a00" (UID: "fb753653-a7e5-46cc-aaf8-d2ce11439a00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:41.095175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.095156 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ce57a0-d866-41c4-840a-cad25c4c46e0-kube-api-access-dxgnc" pod "62ce57a0-d866-41c4-840a-cad25c4c46e0" (UID: "62ce57a0-d866-41c4-840a-cad25c4c46e0"). InnerVolumeSpecName "kube-api-access-dxgnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:41.095508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.095489 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb753653-a7e5-46cc-aaf8-d2ce11439a00-kube-api-access-vf5m2" pod "fb753653-a7e5-46cc-aaf8-d2ce11439a00" (UID: "fb753653-a7e5-46cc-aaf8-d2ce11439a00"). InnerVolumeSpecName "kube-api-access-vf5m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:41.194695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.194662 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxgnc\" (UniqueName: \"kubernetes.io/projected/62ce57a0-d866-41c4-840a-cad25c4c46e0-kube-api-access-dxgnc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:41.194695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.194691 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62ce57a0-d866-41c4-840a-cad25c4c46e0-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:41.194695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.194701 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf5m2\" (UniqueName: \"kubernetes.io/projected/fb753653-a7e5-46cc-aaf8-d2ce11439a00-kube-api-access-vf5m2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:41.194994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.194711 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb753653-a7e5-46cc-aaf8-d2ce11439a00-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:41.564559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.564527 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" Oct 31 21:47:41.564747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.564556 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q" event={"ID":"62ce57a0-d866-41c4-840a-cad25c4c46e0","Type":"ContainerDied","Data":"799332a7cbcbbd1e213b3df68884aa8b30bc9969d47ea696670482ff2d7e4a5f"} Oct 31 21:47:41.564747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.564612 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="799332a7cbcbbd1e213b3df68884aa8b30bc9969d47ea696670482ff2d7e4a5f" Oct 31 21:47:41.566334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.566282 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" Oct 31 21:47:41.566461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.566342 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh" event={"ID":"fb753653-a7e5-46cc-aaf8-d2ce11439a00","Type":"ContainerDied","Data":"5041ad6b5d63edd65ebf846d675c8107c6470cfd4f05de552307487ff481b357"} Oct 31 21:47:41.566461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:41.566387 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5041ad6b5d63edd65ebf846d675c8107c6470cfd4f05de552307487ff481b357" Oct 31 21:47:44.562964 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.562932 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt"] Oct 31 21:47:44.563729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.563707 4294 cpu_manager.go:414] "containerMap: removing container" podUID="62ce57a0-d866-41c4-840a-cad25c4c46e0" containerName="revoke-keys" Oct 31 21:47:44.563729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.563728 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ce57a0-d866-41c4-840a-cad25c4c46e0" containerName="revoke-keys" Oct 31 21:47:44.563877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.563757 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fb753653-a7e5-46cc-aaf8-d2ce11439a00" containerName="revoke-keys" Oct 31 21:47:44.563877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.563765 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb753653-a7e5-46cc-aaf8-d2ce11439a00" containerName="revoke-keys" Oct 31 21:47:44.563982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.563891 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb753653-a7e5-46cc-aaf8-d2ce11439a00" containerName="revoke-keys" Oct 31 21:47:44.563982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.563908 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ce57a0-d866-41c4-840a-cad25c4c46e0" containerName="revoke-keys" Oct 31 21:47:44.569033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.569010 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.594161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.587278 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt"] Oct 31 21:47:44.594161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.590204 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b36e62-openshift-default-dockercfg-thm9x\"" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.670546 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-podinfo\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.670602 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istiod-ca-cert\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.670708 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-data\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.670768 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-token\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.670794 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-socket\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.670884 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djtjf\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-kube-api-access-djtjf\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.670968 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-certs\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.671000 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-envoy\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.697329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.671046 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-credential-socket\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.772256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.772222 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djtjf\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-kube-api-access-djtjf\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.772596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.772568 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-certs\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.772850 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.772832 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-envoy\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.773035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.773002 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-credential-socket\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.773277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.773248 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-podinfo\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.773465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.773450 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istiod-ca-cert\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.773687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.773670 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-data\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.773883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.773854 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-token\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.774051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.773995 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-socket\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.774819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.774743 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-socket\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.775270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.775255 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-credential-socket\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.775573 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.775549 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-data\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.775673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.773072 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-certs\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.776806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.776779 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-envoy\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.777430 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.777404 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istiod-ca-cert\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.779836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.778450 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-podinfo\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.799732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.799697 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-token\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.801537 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.801510 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djtjf\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-kube-api-access-djtjf\") pod \"e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:44.889488 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:44.889404 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:45.231767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.231736 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt"] Oct 31 21:47:45.232388 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:45.232356 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd43c99_e098_4f97_bba2_39fc2ae8b67d.slice/crio-91152e13921120e22764b192d0498ab28cced1c8154e1e984b65c0c6973b605c WatchSource:0}: Error finding container 91152e13921120e22764b192d0498ab28cced1c8154e1e984b65c0c6973b605c: Status 404 returned error can't find the container with id 91152e13921120e22764b192d0498ab28cced1c8154e1e984b65c0c6973b605c Oct 31 21:47:45.235357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.235289 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:45.235469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.235387 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:45.235469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.235414 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:45.600738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.600701 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" event={"ID":"8bd43c99-e098-4f97-bba2-39fc2ae8b67d","Type":"ContainerStarted","Data":"0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9"} Oct 31 21:47:45.601233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.600753 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" event={"ID":"8bd43c99-e098-4f97-bba2-39fc2ae8b67d","Type":"ContainerStarted","Data":"91152e13921120e22764b192d0498ab28cced1c8154e1e984b65c0c6973b605c"} Oct 31 21:47:45.673417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.673365 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podStartSLOduration=1.673351408 podStartE2EDuration="1.673351408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:45.65823352 +0000 UTC m=+1905.783340753" watchObservedRunningTime="2026-10-31 21:47:45.673351408 +0000 UTC m=+1905.798458638" Oct 31 21:47:45.890082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:45.889998 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:46.896227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:46.896198 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:46.969274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:46.969243 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9"] Oct 31 21:47:46.973347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:46.973329 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:46.992409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:46.992378 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9"] Oct 31 21:47:47.036063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.036030 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.036207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.036180 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gzq\" (UniqueName: \"kubernetes.io/projected/7981cce5-e645-44dc-9fdd-6a70355a43fa-kube-api-access-69gzq\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.036294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.036280 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.060943 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.060912 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:47:47.061071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.060915 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:47:47.137498 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.137465 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.137671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.137579 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gzq\" (UniqueName: \"kubernetes.io/projected/7981cce5-e645-44dc-9fdd-6a70355a43fa-kube-api-access-69gzq\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.137671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.137657 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.138161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.138137 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.139970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.139950 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.160785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.160730 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gzq\" (UniqueName: \"kubernetes.io/projected/7981cce5-e645-44dc-9fdd-6a70355a43fa-kube-api-access-69gzq\") pod \"maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.288596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.288558 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:47.606472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.606440 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9"] Oct 31 21:47:47.627950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.627901 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" event={"ID":"7981cce5-e645-44dc-9fdd-6a70355a43fa","Type":"ContainerStarted","Data":"733ca16d637851599af3802976b0688db782fc311ebe08c66a022dfc78a53d65"} Oct 31 21:47:47.628627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.628574 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:47.630475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:47.630453 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:47:48.637813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.637779 4294 generic.go:356] "Generic (PLEG): container finished" podID="7981cce5-e645-44dc-9fdd-6a70355a43fa" containerID="dcafefc2868360a97469dd0636b61db461ac465f798ce44043629ac8f67e378a" exitCode=0 Oct 31 21:47:48.638220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.637853 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" event={"ID":"7981cce5-e645-44dc-9fdd-6a70355a43fa","Type":"ContainerDied","Data":"dcafefc2868360a97469dd0636b61db461ac465f798ce44043629ac8f67e378a"} Oct 31 21:47:48.831920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.831815 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd"] Oct 31 21:47:48.836785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.836752 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.856833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.856139 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-0877f5-openshift-default-dockercfg-qgncs\"" Oct 31 21:47:48.865594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.860931 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd"] Oct 31 21:47:48.973327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973107 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-data\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973196 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-socket\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973243 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9228e-2e5c-4114-8187-423aa04fcae1-istiod-ca-cert\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973321 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-credential-socket\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7r9\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-kube-api-access-gw7r9\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973477 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-token\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973534 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-envoy\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973631 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-podinfo\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:48.973679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:48.973676 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-certs\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.078597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.078558 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-certs\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.079374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.079351 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-data\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.079870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.079853 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-socket\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.080296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.080279 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9228e-2e5c-4114-8187-423aa04fcae1-istiod-ca-cert\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.081181 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.081161 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-credential-socket\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.081993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.081943 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7r9\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-kube-api-access-gw7r9\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.082582 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.082526 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-token\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.083068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.083048 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-envoy\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.083709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.083683 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-podinfo\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.088914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.081870 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-credential-socket\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.089054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.080212 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-socket\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.089146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.079763 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-data\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.089233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.081056 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9228e-2e5c-4114-8187-423aa04fcae1-istiod-ca-cert\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.089322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.079241 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-certs\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.090579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.090558 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-envoy\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.093211 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.093190 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-podinfo\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.109128 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.109098 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-token\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.109652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.109635 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7r9\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-kube-api-access-gw7r9\") pod \"e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.158673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.158625 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:49.491920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.491891 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd"] Oct 31 21:47:49.492959 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:49.492936 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef9228e_2e5c_4114_8187_423aa04fcae1.slice/crio-ba75e6a28971b5c9d17f27db745581952938575a7eeffa2266149171c833aac2 WatchSource:0}: Error finding container ba75e6a28971b5c9d17f27db745581952938575a7eeffa2266149171c833aac2: Status 404 returned error can't find the container with id ba75e6a28971b5c9d17f27db745581952938575a7eeffa2266149171c833aac2 Oct 31 21:47:49.495747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.495706 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:49.495843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.495790 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:49.495907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.495864 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:47:49.652497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.651705 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" event={"ID":"1ef9228e-2e5c-4114-8187-423aa04fcae1","Type":"ContainerStarted","Data":"c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e"} Oct 31 21:47:49.652497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.651774 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" event={"ID":"1ef9228e-2e5c-4114-8187-423aa04fcae1","Type":"ContainerStarted","Data":"ba75e6a28971b5c9d17f27db745581952938575a7eeffa2266149171c833aac2"} Oct 31 21:47:49.712554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.712025 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podStartSLOduration=1.712004969 podStartE2EDuration="1.712004969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:49.702899195 +0000 UTC m=+1909.828006442" watchObservedRunningTime="2026-10-31 21:47:49.712004969 +0000 UTC m=+1909.837112199" Oct 31 21:47:49.832449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.832422 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:49.939892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.939859 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg"] Oct 31 21:47:49.940759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.940736 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7981cce5-e645-44dc-9fdd-6a70355a43fa" containerName="revoke-keys" Oct 31 21:47:49.940822 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.940764 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7981cce5-e645-44dc-9fdd-6a70355a43fa" containerName="revoke-keys" Oct 31 21:47:49.940947 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.940934 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7981cce5-e645-44dc-9fdd-6a70355a43fa" containerName="revoke-keys" Oct 31 21:47:49.945005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.944984 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:49.961883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.961854 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg"] Oct 31 21:47:49.962199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.962180 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b36e62-dockercfg-vw8g4\"" Oct 31 21:47:49.965702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:49.965677 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b36e62\"" Oct 31 21:47:50.017387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.006369 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7981cce5-e645-44dc-9fdd-6a70355a43fa-kube-api-access-69gzq\" (UniqueName: \"kubernetes.io/projected/7981cce5-e645-44dc-9fdd-6a70355a43fa-kube-api-access-69gzq\") pod \"7981cce5-e645-44dc-9fdd-6a70355a43fa\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " Oct 31 21:47:50.017387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.009399 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-xks-ca\") pod \"7981cce5-e645-44dc-9fdd-6a70355a43fa\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " Oct 31 21:47:50.017387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.009504 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-openshift-ca\") pod \"7981cce5-e645-44dc-9fdd-6a70355a43fa\" (UID: \"7981cce5-e645-44dc-9fdd-6a70355a43fa\") " Oct 31 21:47:50.024372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.017955 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-openshift-ca" pod "7981cce5-e645-44dc-9fdd-6a70355a43fa" (UID: "7981cce5-e645-44dc-9fdd-6a70355a43fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:47:50.133144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.133042 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9889bb54-d360-402e-83e8-e175c421291c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:50.133370 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.133349 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhd5\" (UniqueName: \"kubernetes.io/projected/9889bb54-d360-402e-83e8-e175c421291c-kube-api-access-ldhd5\") pod \"payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:50.133491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.133476 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:50.171326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.166662 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7981cce5-e645-44dc-9fdd-6a70355a43fa-kube-api-access-69gzq" pod "7981cce5-e645-44dc-9fdd-6a70355a43fa" (UID: "7981cce5-e645-44dc-9fdd-6a70355a43fa"). InnerVolumeSpecName "kube-api-access-69gzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:47:50.179322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.175956 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-xks-ca" pod "7981cce5-e645-44dc-9fdd-6a70355a43fa" (UID: "7981cce5-e645-44dc-9fdd-6a70355a43fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:47:50.195896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.192050 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:50.212140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.211739 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:50.234507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.234474 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9889bb54-d360-402e-83e8-e175c421291c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:50.234664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.234649 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhd5\" (UniqueName: \"kubernetes.io/projected/9889bb54-d360-402e-83e8-e175c421291c-kube-api-access-ldhd5\") pod \"payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:50.234865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.234845 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69gzq\" (UniqueName: \"kubernetes.io/projected/7981cce5-e645-44dc-9fdd-6a70355a43fa-kube-api-access-69gzq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:50.234865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.234860 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7981cce5-e645-44dc-9fdd-6a70355a43fa-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:47:50.235532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.235504 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9889bb54-d360-402e-83e8-e175c421291c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:50.264693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.264657 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhd5\" (UniqueName: \"kubernetes.io/projected/9889bb54-d360-402e-83e8-e175c421291c-kube-api-access-ldhd5\") pod \"payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:50.266459 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.266437 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:50.605715 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:47:50.605673 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9889bb54_d360_402e_83e8_e175c421291c.slice/crio-f68182762785808c5aae8f48fb91ba19b82132dfe2bdca1f961d9fd32417fcd6 WatchSource:0}: Error finding container f68182762785808c5aae8f48fb91ba19b82132dfe2bdca1f961d9fd32417fcd6: Status 404 returned error can't find the container with id f68182762785808c5aae8f48fb91ba19b82132dfe2bdca1f961d9fd32417fcd6 Oct 31 21:47:50.606686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.606590 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg"] Oct 31 21:47:50.661044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.661011 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" event={"ID":"9889bb54-d360-402e-83e8-e175c421291c","Type":"ContainerStarted","Data":"f68182762785808c5aae8f48fb91ba19b82132dfe2bdca1f961d9fd32417fcd6"} Oct 31 21:47:50.662648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.662627 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" Oct 31 21:47:50.662648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.662635 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9" event={"ID":"7981cce5-e645-44dc-9fdd-6a70355a43fa","Type":"ContainerDied","Data":"733ca16d637851599af3802976b0688db782fc311ebe08c66a022dfc78a53d65"} Oct 31 21:47:50.662818 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.662674 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="733ca16d637851599af3802976b0688db782fc311ebe08c66a022dfc78a53d65" Oct 31 21:47:50.663106 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.663079 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:50.664209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:50.664189 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:47:51.672984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:51.672943 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" event={"ID":"9889bb54-d360-402e-83e8-e175c421291c","Type":"ContainerStarted","Data":"3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc"} Oct 31 21:47:51.673491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:51.673036 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:47:51.715969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:47:51.715918 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" podStartSLOduration=2.715904216 podStartE2EDuration="2.715904216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:47:51.712868802 +0000 UTC m=+1911.837976036" watchObservedRunningTime="2026-10-31 21:47:51.715904216 +0000 UTC m=+1911.841011473" Oct 31 21:48:00.594138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.594101 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd"] Oct 31 21:48:00.607828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.607797 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.619562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.619528 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd"] Oct 31 21:48:00.628711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.628658 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e791\"/\"kube-root-ca.crt\"" Oct 31 21:48:00.634283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.634258 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e791\"/\"openshift-service-ca.crt\"" Oct 31 21:48:00.634433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.634320 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e791\"/\"default-dockercfg-zvjsx\"" Oct 31 21:48:00.634433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.634343 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-f1e791\"/\"auth-test-model-f1e791-kserve-self-signed-certs\"" Oct 31 21:48:00.698547 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.698517 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-tmp-dir\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.698714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.698671 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-dshm\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.698714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.698707 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-home\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.698840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.698814 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drd5w\" (UniqueName: \"kubernetes.io/projected/a78f31ce-8e56-4e08-8319-a635ca291530-kube-api-access-drd5w\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.698890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.698872 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-model-cache\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.698939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.698917 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78f31ce-8e56-4e08-8319-a635ca291530-tls-certs\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.799977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.799940 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-home\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800025 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drd5w\" (UniqueName: \"kubernetes.io/projected/a78f31ce-8e56-4e08-8319-a635ca291530-kube-api-access-drd5w\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800065 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-model-cache\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800104 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78f31ce-8e56-4e08-8319-a635ca291530-tls-certs\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800170 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-tmp-dir\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800362 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-dshm\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800417 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-home\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800645 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-tmp-dir\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.800865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.800839 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-model-cache\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.802717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.802696 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-dshm\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.803336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.803290 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78f31ce-8e56-4e08-8319-a635ca291530-tls-certs\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.824504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.824480 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drd5w\" (UniqueName: \"kubernetes.io/projected/a78f31ce-8e56-4e08-8319-a635ca291530-kube-api-access-drd5w\") pod \"auth-test-model-f1e791-kserve-7f8855758f-sccfd\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.921989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.921655 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:00.942151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.942106 4294 scope.go:122] "RemoveContainer" containerID="89ed7b215920f1652cc652f8902c28c6883c6b567b94bded28b086b0637d34be" Oct 31 21:48:00.989184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:00.989071 4294 scope.go:122] "RemoveContainer" containerID="a35bca9ac7345b8c1e055895fff5eaefcc49891453e21722e627540ae9b34c6b" Oct 31 21:48:01.000704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.000660 4294 scope.go:122] "RemoveContainer" containerID="3760c0678ebfefbca5699a5e8605ac8e915f5c05791df8d953733a12a1a8fc94" Oct 31 21:48:01.027247 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.026889 4294 scope.go:122] "RemoveContainer" containerID="af7127dc41d80fdfcd8d2b5c0a4464dfc3c2179a5c7f4f799eaa2573b5d539ed" Oct 31 21:48:01.042620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.042486 4294 scope.go:122] "RemoveContainer" containerID="ff80d410815a4efc4cfe15b1defeb8b46d6a6ffec2e473b149b844ac1463a84f" Oct 31 21:48:01.293138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.293105 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd"] Oct 31 21:48:01.299095 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:48:01.299059 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda78f31ce_8e56_4e08_8319_a635ca291530.slice/crio-454ad0edb11bcad351419c37d34c84f74cda5abf1109107c38ed5b5b11e39871 WatchSource:0}: Error finding container 454ad0edb11bcad351419c37d34c84f74cda5abf1109107c38ed5b5b11e39871: Status 404 returned error can't find the container with id 454ad0edb11bcad351419c37d34c84f74cda5abf1109107c38ed5b5b11e39871 Oct 31 21:48:01.423325 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:48:01.373820 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:48:01.718895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.718809 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9"] Oct 31 21:48:01.722555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.722528 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-9c8b33-xdfm9"] Oct 31 21:48:01.766451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.766414 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" event={"ID":"a78f31ce-8e56-4e08-8319-a635ca291530","Type":"ContainerStarted","Data":"d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616"} Oct 31 21:48:01.766595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.766459 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" event={"ID":"a78f31ce-8e56-4e08-8319-a635ca291530","Type":"ContainerStarted","Data":"454ad0edb11bcad351419c37d34c84f74cda5abf1109107c38ed5b5b11e39871"} Oct 31 21:48:01.810550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.810470 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" podStartSLOduration=1.810451603 podStartE2EDuration="1.810451603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:01.807763618 +0000 UTC m=+1921.932870849" watchObservedRunningTime="2026-10-31 21:48:01.810451603 +0000 UTC m=+1921.935558834" Oct 31 21:48:01.924147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.923088 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:01.985493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:01.985457 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7981cce5-e645-44dc-9fdd-6a70355a43fa" path="/var/lib/kubelet/pods/7981cce5-e645-44dc-9fdd-6a70355a43fa/volumes" Oct 31 21:48:02.683670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:02.683639 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 21:48:04.423001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.422970 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs"] Oct 31 21:48:04.428237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.428219 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.448187 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.448163 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs"] Oct 31 21:48:04.461310 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.461267 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:48:04.461432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.461356 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:48:04.497641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.497608 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.497776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.497754 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w6bd\" (UniqueName: \"kubernetes.io/projected/40c864bc-4697-4f91-9508-b4f25aea18b4-kube-api-access-7w6bd\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.497970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.497949 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.598930 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.598896 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w6bd\" (UniqueName: \"kubernetes.io/projected/40c864bc-4697-4f91-9508-b4f25aea18b4-kube-api-access-7w6bd\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.599120 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.599093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.599186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.599168 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.599839 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.599817 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.601347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.601324 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.622341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.622287 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w6bd\" (UniqueName: \"kubernetes.io/projected/40c864bc-4697-4f91-9508-b4f25aea18b4-kube-api-access-7w6bd\") pod \"maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:04.742088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:04.742059 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:05.047786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:05.047759 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs"] Oct 31 21:48:05.048266 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:48:05.048242 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c864bc_4697_4f91_9508_b4f25aea18b4.slice/crio-37e455aae5fe26f8485a3a7a2bba9746034cd6c2363f04b60a28e6921b53a83b WatchSource:0}: Error finding container 37e455aae5fe26f8485a3a7a2bba9746034cd6c2363f04b60a28e6921b53a83b: Status 404 returned error can't find the container with id 37e455aae5fe26f8485a3a7a2bba9746034cd6c2363f04b60a28e6921b53a83b Oct 31 21:48:05.803793 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:05.803754 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" event={"ID":"40c864bc-4697-4f91-9508-b4f25aea18b4","Type":"ContainerStarted","Data":"a95c1d9a153a2284039a4079a36719dcaadb776773c0a282233912635110567f"} Oct 31 21:48:05.804181 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:05.803816 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" event={"ID":"40c864bc-4697-4f91-9508-b4f25aea18b4","Type":"ContainerStarted","Data":"37e455aae5fe26f8485a3a7a2bba9746034cd6c2363f04b60a28e6921b53a83b"} Oct 31 21:48:05.845640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:05.845585 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" podStartSLOduration=1.845571554 podStartE2EDuration="1.845571554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:05.84362734 +0000 UTC m=+1925.968734571" watchObservedRunningTime="2026-10-31 21:48:05.845571554 +0000 UTC m=+1925.970678785" Oct 31 21:48:06.812823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:06.812793 4294 generic.go:356] "Generic (PLEG): container finished" podID="40c864bc-4697-4f91-9508-b4f25aea18b4" containerID="a95c1d9a153a2284039a4079a36719dcaadb776773c0a282233912635110567f" exitCode=0 Oct 31 21:48:06.813212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:06.812881 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" event={"ID":"40c864bc-4697-4f91-9508-b4f25aea18b4","Type":"ContainerDied","Data":"a95c1d9a153a2284039a4079a36719dcaadb776773c0a282233912635110567f"} Oct 31 21:48:08.119856 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.111511 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:08.184745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.184709 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-xks-ca\") pod \"40c864bc-4697-4f91-9508-b4f25aea18b4\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " Oct 31 21:48:08.184914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.184836 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40c864bc-4697-4f91-9508-b4f25aea18b4-kube-api-access-7w6bd\" (UniqueName: \"kubernetes.io/projected/40c864bc-4697-4f91-9508-b4f25aea18b4-kube-api-access-7w6bd\") pod \"40c864bc-4697-4f91-9508-b4f25aea18b4\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " Oct 31 21:48:08.185056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.185040 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-openshift-ca\") pod \"40c864bc-4697-4f91-9508-b4f25aea18b4\" (UID: \"40c864bc-4697-4f91-9508-b4f25aea18b4\") " Oct 31 21:48:08.185708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.185671 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-openshift-ca" pod "40c864bc-4697-4f91-9508-b4f25aea18b4" (UID: "40c864bc-4697-4f91-9508-b4f25aea18b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:08.187911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.187885 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-xks-ca" pod "40c864bc-4697-4f91-9508-b4f25aea18b4" (UID: "40c864bc-4697-4f91-9508-b4f25aea18b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:48:08.188018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.187975 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40c864bc-4697-4f91-9508-b4f25aea18b4-kube-api-access-7w6bd" pod "40c864bc-4697-4f91-9508-b4f25aea18b4" (UID: "40c864bc-4697-4f91-9508-b4f25aea18b4"). InnerVolumeSpecName "kube-api-access-7w6bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:08.286861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.286814 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:08.286861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.286856 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40c864bc-4697-4f91-9508-b4f25aea18b4-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:08.287091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.286870 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w6bd\" (UniqueName: \"kubernetes.io/projected/40c864bc-4697-4f91-9508-b4f25aea18b4-kube-api-access-7w6bd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:08.828249 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.828216 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" Oct 31 21:48:08.828415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.828246 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs" event={"ID":"40c864bc-4697-4f91-9508-b4f25aea18b4","Type":"ContainerDied","Data":"37e455aae5fe26f8485a3a7a2bba9746034cd6c2363f04b60a28e6921b53a83b"} Oct 31 21:48:08.828415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:08.828291 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37e455aae5fe26f8485a3a7a2bba9746034cd6c2363f04b60a28e6921b53a83b" Oct 31 21:48:09.751618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.751585 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z"] Oct 31 21:48:09.752160 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.752146 4294 cpu_manager.go:414] "containerMap: removing container" podUID="40c864bc-4697-4f91-9508-b4f25aea18b4" containerName="revoke-keys" Oct 31 21:48:09.752202 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.752162 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40c864bc-4697-4f91-9508-b4f25aea18b4" containerName="revoke-keys" Oct 31 21:48:09.752284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.752275 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="40c864bc-4697-4f91-9508-b4f25aea18b4" containerName="revoke-keys" Oct 31 21:48:09.756176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.756159 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.793415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.793378 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z"] Oct 31 21:48:09.909653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.909605 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e606df7-cf70-401f-af4c-0639a8bdaff0-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.909824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.909665 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.909908 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.909855 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.910000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.909979 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.910045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.910024 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnw2\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-kube-api-access-ljnw2\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.910099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.910064 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.910264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.910243 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.910330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.910294 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:09.910375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:09.910366 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012006 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012060 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e606df7-cf70-401f-af4c-0639a8bdaff0-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012177 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012359 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012458 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012502 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012508 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnw2\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-kube-api-access-ljnw2\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012562 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012645 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012673 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012694 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012724 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e606df7-cf70-401f-af4c-0639a8bdaff0-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012876 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012793 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.012964 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.012943 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.014805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.014777 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.015019 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.015004 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.036583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.036552 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.036694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.036658 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnw2\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-kube-api-access-ljnw2\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.073067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.073041 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:10.399774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.399741 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z"] Oct 31 21:48:10.400818 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:48:10.400788 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e606df7_cf70_401f_af4c_0639a8bdaff0.slice/crio-2ced035bf46c09528d66e22b8275f2fb6adfaa27acec4fcc5fbcec920f124675 WatchSource:0}: Error finding container 2ced035bf46c09528d66e22b8275f2fb6adfaa27acec4fcc5fbcec920f124675: Status 404 returned error can't find the container with id 2ced035bf46c09528d66e22b8275f2fb6adfaa27acec4fcc5fbcec920f124675 Oct 31 21:48:10.403742 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.403705 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:10.403833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.403779 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:10.403833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.403806 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:10.858637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.858595 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" event={"ID":"3e606df7-cf70-401f-af4c-0639a8bdaff0","Type":"ContainerStarted","Data":"ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813"} Oct 31 21:48:10.859069 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.858664 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" event={"ID":"3e606df7-cf70-401f-af4c-0639a8bdaff0","Type":"ContainerStarted","Data":"2ced035bf46c09528d66e22b8275f2fb6adfaa27acec4fcc5fbcec920f124675"} Oct 31 21:48:10.931331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.927688 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:10.964328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:10.963719 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podStartSLOduration=1.9636953849999998 podStartE2EDuration="1.963695385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:10.906007249 +0000 UTC m=+1931.031114484" watchObservedRunningTime="2026-10-31 21:48:10.963695385 +0000 UTC m=+1931.088802617" Oct 31 21:48:11.056328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:11.055887 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:11.078581 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:11.078543 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:11.083342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:11.083150 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 21:48:11.083342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:11.083208 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 21:48:11.909950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:11.909918 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:48:12.077331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:12.075679 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 21:48:12.077331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:12.075740 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 21:48:13.077380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:13.075710 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 21:48:13.077380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:13.075785 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 21:48:14.078347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:14.075735 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 21:48:14.078347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:14.075805 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 21:48:15.095812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:15.095772 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 21:48:15.096479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:15.096441 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 21:48:15.945513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:15.945483 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz"] Oct 31 21:48:15.945710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:15.945673 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" containerID="cri-o://e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384" gracePeriod=30 Oct 31 21:48:16.082338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:16.078091 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 21:48:16.082338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:16.078154 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 21:48:17.073597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:17.073522 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" start-of-body= Oct 31 21:48:17.074089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:17.073620 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.159:15021/healthz/ready\": dial tcp 10.134.0.159:15021: connect: connection refused" Oct 31 21:48:18.078863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:18.078829 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:18.079389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:18.079369 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:18.080233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:18.080210 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:48:20.082875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:20.082838 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:48:20.083294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:20.082900 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:48:21.223392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.223324 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:48:21.350242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350208 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istiod-ca-cert\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350272 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-envoy\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350295 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-certs\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350440 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-token\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-token\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350463 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-kube-api-access-bd4wn\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-kube-api-access-bd4wn\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350506 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-data\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350632 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-certs" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:21.350678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350655 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istiod-ca-cert" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:21.350845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350657 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-podinfo\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350719 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-credential-socket\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.350845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350773 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-data" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:21.350845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350793 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-socket\") pod \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\" (UID: \"abebb1e3-0deb-4ca7-abdc-0c609e75fbed\") " Oct 31 21:48:21.351067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.350948 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-credential-socket" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:21.351133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.351081 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-socket" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:21.351708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.351670 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.351708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.351705 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.351884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.351719 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.351884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.351734 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.351884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.351747 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.352636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.352608 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-envoy" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:21.352721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.352634 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-token" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:21.352775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.352721 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-kube-api-access-bd4wn" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "kube-api-access-bd4wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:21.352858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.352839 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-podinfo" pod "abebb1e3-0deb-4ca7-abdc-0c609e75fbed" (UID: "abebb1e3-0deb-4ca7-abdc-0c609e75fbed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:48:21.452766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.452731 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.452766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.452761 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.452766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.452771 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd4wn\" (UniqueName: \"kubernetes.io/projected/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-kube-api-access-bd4wn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.453129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.452781 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abebb1e3-0deb-4ca7-abdc-0c609e75fbed-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:21.984192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.984163 4294 generic.go:356] "Generic (PLEG): container finished" podID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerID="e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384" exitCode=0 Oct 31 21:48:21.984406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.984255 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" Oct 31 21:48:21.985535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.985508 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" event={"ID":"abebb1e3-0deb-4ca7-abdc-0c609e75fbed","Type":"ContainerDied","Data":"e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384"} Oct 31 21:48:21.985660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.985552 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz" event={"ID":"abebb1e3-0deb-4ca7-abdc-0c609e75fbed","Type":"ContainerDied","Data":"c2350d2d16c4586cdbfbf55506262a19fb16b188de0885d7dbf6fdc7e1c90470"} Oct 31 21:48:21.985660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:21.985579 4294 scope.go:122] "RemoveContainer" containerID="e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384" Oct 31 21:48:22.000275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:22.000250 4294 scope.go:122] "RemoveContainer" containerID="e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384" Oct 31 21:48:22.000813 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:48:22.000790 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384\": container with ID starting with e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384 not found: ID does not exist" containerID="e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384" Oct 31 21:48:22.000872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:22.000819 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384"} err="failed to get container status \"e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384\": rpc error: code = NotFound desc = could not find container \"e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384\": container with ID starting with e1b1ce921583b95ae35539a8e49362f2ec170aea4c7ea3a2a08017377ab20384 not found: ID does not exist" Oct 31 21:48:22.051911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:22.051881 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz"] Oct 31 21:48:22.063936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:22.063903 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-6ljcz"] Oct 31 21:48:24.030700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.030662 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" path="/var/lib/kubelet/pods/abebb1e3-0deb-4ca7-abdc-0c609e75fbed/volumes" Oct 31 21:48:24.470695 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.470595 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs"] Oct 31 21:48:24.475171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.475147 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b01ff6-wz4zs"] Oct 31 21:48:24.746276 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.746237 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r"] Oct 31 21:48:24.747155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.747133 4294 cpu_manager.go:414] "containerMap: removing container" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" Oct 31 21:48:24.747213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.747158 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" Oct 31 21:48:24.747384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.747369 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="abebb1e3-0deb-4ca7-abdc-0c609e75fbed" containerName="istio-proxy" Oct 31 21:48:24.751201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.751181 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.771733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.771686 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f77b4\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-kube-api-access-f77b4\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.771896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.771743 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.771896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.771794 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.771896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.771826 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.771896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.771869 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.772134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.771975 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.772134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.772065 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.772134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.772096 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.772134 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.772123 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.773210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.773183 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r"] Oct 31 21:48:24.873091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873050 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873178 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873264 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873312 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873341 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873424 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f77b4\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-kube-api-access-f77b4\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873454 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873506 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873544 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873638 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873902 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873774 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-certs\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.873970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873944 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-data\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.874027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.873970 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-credential-socket\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.874081 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.874060 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istiod-ca-cert\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.875675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.875654 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-envoy\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.875817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.875802 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-podinfo\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.896447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.896418 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-token\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:24.896542 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:24.896464 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f77b4\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-kube-api-access-f77b4\") pod \"e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:25.065188 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:25.065109 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:25.389296 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:48:25.389266 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7ffa0f_8116_4c06_a302_cfd9071d85bf.slice/crio-b74df6d91af0f0f30264addd6b87898491b359e541a2ea75d49cdc64d074e71d WatchSource:0}: Error finding container b74df6d91af0f0f30264addd6b87898491b359e541a2ea75d49cdc64d074e71d: Status 404 returned error can't find the container with id b74df6d91af0f0f30264addd6b87898491b359e541a2ea75d49cdc64d074e71d Oct 31 21:48:25.391914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:25.391880 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:25.392022 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:25.391956 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:25.392022 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:25.391987 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:25.396885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:25.396856 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r"] Oct 31 21:48:26.012777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:26.012739 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40c864bc-4697-4f91-9508-b4f25aea18b4" path="/var/lib/kubelet/pods/40c864bc-4697-4f91-9508-b4f25aea18b4/volumes" Oct 31 21:48:26.050704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:26.050671 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" event={"ID":"3b7ffa0f-8116-4c06-a302-cfd9071d85bf","Type":"ContainerStarted","Data":"90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2"} Oct 31 21:48:26.050870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:26.050723 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" event={"ID":"3b7ffa0f-8116-4c06-a302-cfd9071d85bf","Type":"ContainerStarted","Data":"b74df6d91af0f0f30264addd6b87898491b359e541a2ea75d49cdc64d074e71d"} Oct 31 21:48:26.065369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:26.065331 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:26.067316 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:26.067281 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 21:48:26.067449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:26.067350 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 21:48:26.132667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:26.132596 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podStartSLOduration=2.132581644 podStartE2EDuration="2.132581644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:26.120278456 +0000 UTC m=+1946.245385701" watchObservedRunningTime="2026-10-31 21:48:26.132581644 +0000 UTC m=+1946.257688874" Oct 31 21:48:27.065897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:27.065857 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 21:48:27.066263 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:27.065924 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 21:48:28.065739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:28.065700 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 21:48:28.065910 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:28.065761 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 21:48:29.065498 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:29.065462 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 21:48:29.065687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:29.065522 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 21:48:30.066074 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:30.066040 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 21:48:30.066572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:30.066100 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 21:48:31.065627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:31.065592 4294 patch_prober.go:29] interesting pod/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body= Oct 31 21:48:31.065809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:31.065653 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" Oct 31 21:48:31.311825 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:48:31.311785 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:48:32.451124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:32.451084 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:32.451694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:32.451529 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:32.452396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:32.452373 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:48:32.512249 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:32.512205 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf"] Oct 31 21:48:32.512490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:32.512463 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" containerID="cri-o://8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa" gracePeriod=30 Oct 31 21:48:33.227794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.227754 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk"] Oct 31 21:48:33.233417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.233395 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.248105 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.248080 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:48:33.250712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.250687 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk"] Oct 31 21:48:33.257627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.257604 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:48:33.332545 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.332514 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.332696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.332554 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.332696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.332655 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c52nj\" (UniqueName: \"kubernetes.io/projected/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-kube-api-access-c52nj\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.434270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.434242 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c52nj\" (UniqueName: \"kubernetes.io/projected/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-kube-api-access-c52nj\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.434448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.434434 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.434498 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.434461 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.435059 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.435042 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.436781 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.436758 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.457440 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.457417 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c52nj\" (UniqueName: \"kubernetes.io/projected/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-kube-api-access-c52nj\") pod \"maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.549888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.549859 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:33.883015 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:48:33.882987 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1e5dbe_843c_48a0_9f0a_8722ce84aa11.slice/crio-2982f09b69a25d4bf48cc3d8b2a134febfe5f0df4dfb7d978e96e70cd2639d41 WatchSource:0}: Error finding container 2982f09b69a25d4bf48cc3d8b2a134febfe5f0df4dfb7d978e96e70cd2639d41: Status 404 returned error can't find the container with id 2982f09b69a25d4bf48cc3d8b2a134febfe5f0df4dfb7d978e96e70cd2639d41 Oct 31 21:48:33.883697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:33.883670 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk"] Oct 31 21:48:34.140385 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.140286 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" event={"ID":"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11","Type":"ContainerStarted","Data":"2982f09b69a25d4bf48cc3d8b2a134febfe5f0df4dfb7d978e96e70cd2639d41"} Oct 31 21:48:34.501470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.501430 4294 patch_prober.go:29] interesting pod/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:48:34.501914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.501501 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:48:34.548516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.548478 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82"] Oct 31 21:48:34.555387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.555360 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.568076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.568047 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82"] Oct 31 21:48:34.614897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.614869 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-fb32d4-openshift-default-dockercfg-2vbkm\"" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6dff6d-171c-4611-924d-034020433dd6-istio-podinfo\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675461 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-data\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675532 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4hf\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-kube-api-access-wd4hf\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675618 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6dff6d-171c-4611-924d-034020433dd6-istiod-ca-cert\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675718 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-envoy\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675878 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-credential-socket\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675937 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-socket\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.675974 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-certs\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.676324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.676015 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-istio-token\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.779429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.779336 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-credential-socket\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.780326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.780197 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-credential-socket\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.780545 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.780527 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-socket\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.784334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.780941 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-socket\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.784518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.784499 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-certs\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.784954 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.784853 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-certs\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.785091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.784929 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-istio-token\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.785584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.785564 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6dff6d-171c-4611-924d-034020433dd6-istio-podinfo\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.787816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.787796 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-data\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.789726 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.788194 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-data\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.790183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.789896 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4hf\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-kube-api-access-wd4hf\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.791514 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.791494 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6dff6d-171c-4611-924d-034020433dd6-istiod-ca-cert\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.791675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.791658 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6dff6d-171c-4611-924d-034020433dd6-istiod-ca-cert\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.791870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.791855 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-envoy\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.794962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.794941 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-envoy\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.823250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.823217 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-istio-token\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.829422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.829366 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6dff6d-171c-4611-924d-034020433dd6-istio-podinfo\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.833349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.833318 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4hf\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-kube-api-access-wd4hf\") pod \"e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:34.888562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:34.888521 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:35.183611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:35.183514 4294 generic.go:356] "Generic (PLEG): container finished" podID="4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" containerID="6dc99b8a09a28eba945a669648f7fea02fc34613c063b76134be3327a6c90c8c" exitCode=0 Oct 31 21:48:35.184082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:35.184056 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" event={"ID":"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11","Type":"ContainerDied","Data":"6dc99b8a09a28eba945a669648f7fea02fc34613c063b76134be3327a6c90c8c"} Oct 31 21:48:35.299473 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:48:35.299424 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6dff6d_171c_4611_924d_034020433dd6.slice/crio-95e2ac022a059122363cd9b09f7e6ee8d4e30836d23d346b1f41bc0906c6b516 WatchSource:0}: Error finding container 95e2ac022a059122363cd9b09f7e6ee8d4e30836d23d346b1f41bc0906c6b516: Status 404 returned error can't find the container with id 95e2ac022a059122363cd9b09f7e6ee8d4e30836d23d346b1f41bc0906c6b516 Oct 31 21:48:35.301720 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:35.301690 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82"] Oct 31 21:48:35.302032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:35.301997 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:35.302143 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:35.302082 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:35.302209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:35.302159 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:36.192679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.192624 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" event={"ID":"5d6dff6d-171c-4611-924d-034020433dd6","Type":"ContainerStarted","Data":"bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0"} Oct 31 21:48:36.193174 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.192694 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" event={"ID":"5d6dff6d-171c-4611-924d-034020433dd6","Type":"ContainerStarted","Data":"95e2ac022a059122363cd9b09f7e6ee8d4e30836d23d346b1f41bc0906c6b516"} Oct 31 21:48:36.263069 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.263003 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podStartSLOduration=2.262982334 podStartE2EDuration="2.262982334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:36.255718163 +0000 UTC m=+1956.380825395" watchObservedRunningTime="2026-10-31 21:48:36.262982334 +0000 UTC m=+1956.388089564" Oct 31 21:48:36.421914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.421865 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:36.522938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.522904 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-openshift-ca\") pod \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " Oct 31 21:48:36.522938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.522946 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-xks-ca\") pod \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " Oct 31 21:48:36.523213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.523071 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-kube-api-access-c52nj\" (UniqueName: \"kubernetes.io/projected/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-kube-api-access-c52nj\") pod \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\" (UID: \"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11\") " Oct 31 21:48:36.523403 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.523373 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-openshift-ca" pod "4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" (UID: "4b1e5dbe-843c-48a0-9f0a-8722ce84aa11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:36.523787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.523746 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:36.525820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.525792 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-kube-api-access-c52nj" pod "4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" (UID: "4b1e5dbe-843c-48a0-9f0a-8722ce84aa11"). InnerVolumeSpecName "kube-api-access-c52nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:36.525992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.525964 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-xks-ca" pod "4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" (UID: "4b1e5dbe-843c-48a0-9f0a-8722ce84aa11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:48:36.631220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.631154 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:36.631220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.631189 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c52nj\" (UniqueName: \"kubernetes.io/projected/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11-kube-api-access-c52nj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:36.888736 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.888654 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:36.894368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:36.894340 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:37.202655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.202590 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" Oct 31 21:48:37.203001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.202660 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk" event={"ID":"4b1e5dbe-843c-48a0-9f0a-8722ce84aa11","Type":"ContainerDied","Data":"2982f09b69a25d4bf48cc3d8b2a134febfe5f0df4dfb7d978e96e70cd2639d41"} Oct 31 21:48:37.203001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.202703 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2982f09b69a25d4bf48cc3d8b2a134febfe5f0df4dfb7d978e96e70cd2639d41" Oct 31 21:48:37.203209 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.203190 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:37.203846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.203825 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:48:37.785289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.785231 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:48:37.951885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.951846 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-kube-api-access-dwmvq\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-kube-api-access-dwmvq\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.951937 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-credential-socket\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952007 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-certs\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952090 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-data\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952159 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-envoy\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952224 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-token\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-token\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952424 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952291 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-podinfo\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952424 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952380 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-credential-socket" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:37.952424 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952395 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istiod-ca-cert\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.952589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952416 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-data" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:37.952589 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.952485 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-socket\") pod \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\" (UID: \"f4cc3f4e-9300-479c-b129-c0f12eed77f1\") " Oct 31 21:48:37.953341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.953186 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istiod-ca-cert" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:37.953821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.953756 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:37.953821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.953784 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:37.953821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.953799 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:37.954556 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.954512 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-certs" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:37.988535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.974475 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-socket" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:37.989418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.989358 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-envoy" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:48:37.995323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:37.992704 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-podinfo" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:48:38.053405 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.029682 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-kube-api-access-dwmvq" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "kube-api-access-dwmvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:38.053405 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.038667 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-token" pod "f4cc3f4e-9300-479c-b129-c0f12eed77f1" (UID: "f4cc3f4e-9300-479c-b129-c0f12eed77f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:38.082325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.057370 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:38.082325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.057404 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:38.082325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.057416 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4cc3f4e-9300-479c-b129-c0f12eed77f1-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:38.082325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.057428 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:38.082325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.057455 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmvq\" (UniqueName: \"kubernetes.io/projected/f4cc3f4e-9300-479c-b129-c0f12eed77f1-kube-api-access-dwmvq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:38.082325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.057469 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4cc3f4e-9300-479c-b129-c0f12eed77f1-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:38.211095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.211063 4294 generic.go:356] "Generic (PLEG): container finished" podID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerID="8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa" exitCode=0 Oct 31 21:48:38.211630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.211172 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" Oct 31 21:48:38.211630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.211162 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" event={"ID":"f4cc3f4e-9300-479c-b129-c0f12eed77f1","Type":"ContainerDied","Data":"8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa"} Oct 31 21:48:38.211630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.211337 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf" event={"ID":"f4cc3f4e-9300-479c-b129-c0f12eed77f1","Type":"ContainerDied","Data":"c550af9362c07ad83830c617194ee7f717f264d6a82652464e94259da224cbcd"} Oct 31 21:48:38.211630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.211367 4294 scope.go:122] "RemoveContainer" containerID="8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa" Oct 31 21:48:38.222733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.222701 4294 scope.go:122] "RemoveContainer" containerID="854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22" Oct 31 21:48:38.233200 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.233182 4294 scope.go:122] "RemoveContainer" containerID="8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa" Oct 31 21:48:38.233538 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:48:38.233519 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa\": container with ID starting with 8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa not found: ID does not exist" containerID="8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa" Oct 31 21:48:38.233597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.233548 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa"} err="failed to get container status \"8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa\": rpc error: code = NotFound desc = could not find container \"8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa\": container with ID starting with 8006b2325ec14155da1417cd3d79361a169b3b56d92354be3ca336511ad98ffa not found: ID does not exist" Oct 31 21:48:38.233597 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.233564 4294 scope.go:122] "RemoveContainer" containerID="854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22" Oct 31 21:48:38.233846 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:48:38.233826 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22\": container with ID starting with 854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22 not found: ID does not exist" containerID="854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22" Oct 31 21:48:38.233916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.233864 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22"} err="failed to get container status \"854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22\": rpc error: code = NotFound desc = could not find container \"854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22\": container with ID starting with 854b08587d33a29e49393388c644e0e4aeeb4cad780aad6cadb065b1d1736c22 not found: ID does not exist" Oct 31 21:48:38.270046 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.270018 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf"] Oct 31 21:48:38.282042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.282004 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b01ff6-openshift-default-5f7d9bdd69-6vgsf"] Oct 31 21:48:38.785418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.785375 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw"] Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.786941 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.786968 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.786984 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.786992 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.787102 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" containerName="revoke-keys" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.787119 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" containerName="revoke-keys" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.787599 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" containerName="revoke-keys" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.787639 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.787650 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" containerName="istio-proxy" Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.796778 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw"] Oct 31 21:48:38.808335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.796934 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.833411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.816969 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-5237cd-openshift-default-dockercfg-76npr\"" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.988900 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swcsd\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-kube-api-access-swcsd\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.988954 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-token\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.989078 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istiod-ca-cert\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.989131 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-envoy\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.989204 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-podinfo\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.989233 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-data\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.989317 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-socket\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.989351 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-credential-socket\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:38.995325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:38.989438 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-certs\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091229 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istiod-ca-cert\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091343 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-envoy\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091420 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-podinfo\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091453 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-data\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091522 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-socket\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091554 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-credential-socket\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091646 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-certs\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091714 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swcsd\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-kube-api-access-swcsd\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.092401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.091748 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-token\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.093000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.092985 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istiod-ca-cert\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.093473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.093447 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-socket\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.093921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.093900 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-credential-socket\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.094519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.094493 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-certs\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.099063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.095601 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-data\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.108425 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.106906 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-envoy\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.125511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.124866 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-podinfo\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.136328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.133920 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-token\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.136328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.134582 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swcsd\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-kube-api-access-swcsd\") pod \"e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.428623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.428538 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:39.769229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.769190 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw"] Oct 31 21:48:39.772524 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:48:39.772492 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fbfdbbd_4a73_4b68_86c9_ac418fceb78e.slice/crio-4c840822488fa8ad355fcefbee64d20d8e6eb6df5a55f8c344e5dce514f59a2f WatchSource:0}: Error finding container 4c840822488fa8ad355fcefbee64d20d8e6eb6df5a55f8c344e5dce514f59a2f: Status 404 returned error can't find the container with id 4c840822488fa8ad355fcefbee64d20d8e6eb6df5a55f8c344e5dce514f59a2f Oct 31 21:48:39.775752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.775712 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:39.775841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.775800 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:39.775904 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.775841 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:39.990126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:39.989955 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4cc3f4e-9300-479c-b129-c0f12eed77f1" path="/var/lib/kubelet/pods/f4cc3f4e-9300-479c-b129-c0f12eed77f1/volumes" Oct 31 21:48:40.250330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:40.249492 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" event={"ID":"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e","Type":"ContainerStarted","Data":"771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e"} Oct 31 21:48:40.250330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:40.249561 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" event={"ID":"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e","Type":"ContainerStarted","Data":"4c840822488fa8ad355fcefbee64d20d8e6eb6df5a55f8c344e5dce514f59a2f"} Oct 31 21:48:40.338325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:40.336190 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podStartSLOduration=2.336171369 podStartE2EDuration="2.336171369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:48:40.323030666 +0000 UTC m=+1960.448137915" watchObservedRunningTime="2026-10-31 21:48:40.336171369 +0000 UTC m=+1960.461278601" Oct 31 21:48:40.429577 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:40.429538 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:40.439660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:40.439629 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:41.260193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:41.260159 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:41.261200 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:41.261179 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:48:43.766756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:43.766718 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx"] Oct 31 21:48:48.670433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:48.670399 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:48:48.670809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:48.670453 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:48:48.769264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:48.767491 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" podUID="34b3dab4-e9e1-42c6-baa9-1ad202b611eb" containerName="payload-pre-processing" containerID="cri-o://d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87" gracePeriod=25 Oct 31 21:48:49.046483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.046458 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:48:49.145330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.145289 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-plugins-config-volume\") pod \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " Oct 31 21:48:49.145564 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.145550 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-kube-api-access-tvx4x\" (UniqueName: \"kubernetes.io/projected/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-kube-api-access-tvx4x\") pod \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\" (UID: \"34b3dab4-e9e1-42c6-baa9-1ad202b611eb\") " Oct 31 21:48:49.145653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.145628 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-plugins-config-volume" pod "34b3dab4-e9e1-42c6-baa9-1ad202b611eb" (UID: "34b3dab4-e9e1-42c6-baa9-1ad202b611eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:48:49.146008 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.145987 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:49.148077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.148048 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-kube-api-access-tvx4x" pod "34b3dab4-e9e1-42c6-baa9-1ad202b611eb" (UID: "34b3dab4-e9e1-42c6-baa9-1ad202b611eb"). InnerVolumeSpecName "kube-api-access-tvx4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:48:49.247471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.247440 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvx4x\" (UniqueName: \"kubernetes.io/projected/34b3dab4-e9e1-42c6-baa9-1ad202b611eb-kube-api-access-tvx4x\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:48:49.353094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.353061 4294 generic.go:356] "Generic (PLEG): container finished" podID="34b3dab4-e9e1-42c6-baa9-1ad202b611eb" containerID="d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87" exitCode=0 Oct 31 21:48:49.353282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.353108 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" Oct 31 21:48:49.353282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.353140 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" event={"ID":"34b3dab4-e9e1-42c6-baa9-1ad202b611eb","Type":"ContainerDied","Data":"d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87"} Oct 31 21:48:49.353282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.353189 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx" event={"ID":"34b3dab4-e9e1-42c6-baa9-1ad202b611eb","Type":"ContainerDied","Data":"744b39c43c3489374d6f45941c793594b9e44daedca292d3e11a8c61043b8fa0"} Oct 31 21:48:49.353282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.353211 4294 scope.go:122] "RemoveContainer" containerID="d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87" Oct 31 21:48:49.354966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.354935 4294 generic.go:356] "Generic (PLEG): container finished" podID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerID="0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9" exitCode=0 Oct 31 21:48:49.355052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.354965 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" event={"ID":"8bd43c99-e098-4f97-bba2-39fc2ae8b67d","Type":"ContainerDied","Data":"0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9"} Oct 31 21:48:49.355386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.355373 4294 scope.go:122] "RemoveContainer" containerID="0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9" Oct 31 21:48:49.356884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.356851 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:49.357000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.356925 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:49.357000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.356960 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:48:49.364409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.364389 4294 scope.go:122] "RemoveContainer" containerID="d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87" Oct 31 21:48:49.364698 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:48:49.364680 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87\": container with ID starting with d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87 not found: ID does not exist" containerID="d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87" Oct 31 21:48:49.364763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.364710 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87"} err="failed to get container status \"d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87\": rpc error: code = NotFound desc = could not find container \"d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87\": container with ID starting with d89613089b5b14b718f8ed459db700c596266dd1acd60bd6b4ab79dd78fa6b87 not found: ID does not exist" Oct 31 21:48:49.406468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.406441 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx"] Oct 31 21:48:49.410806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:49.410774 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ad121b-7dcf49447-m9fhx"] Oct 31 21:48:50.008558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:50.008525 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b3dab4-e9e1-42c6-baa9-1ad202b611eb" path="/var/lib/kubelet/pods/34b3dab4-e9e1-42c6-baa9-1ad202b611eb/volumes" Oct 31 21:48:50.365745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:50.365660 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" event={"ID":"8bd43c99-e098-4f97-bba2-39fc2ae8b67d","Type":"ContainerStarted","Data":"208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8"} Oct 31 21:48:50.891326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:50.890740 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:48:50.891326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:50.890869 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 31 21:48:50.891326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:50.890920 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 31 21:48:51.896330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:51.895397 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 31 21:48:51.896330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:51.895467 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 31 21:48:52.895102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:52.895068 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 31 21:48:52.895510 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:52.895464 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 31 21:48:53.891698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:53.891658 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 31 21:48:53.892333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:53.892279 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 31 21:48:54.893326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:54.891639 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body= Oct 31 21:48:54.893326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:54.891700 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" Oct 31 21:48:55.894033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:55.894003 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:48:55.894519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:55.894501 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:48:55.895167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:48:55.895148 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 21:49:01.313056 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:49:01.313014 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:49:05.567000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:05.566967 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk"] Oct 31 21:49:05.579754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:05.579726 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ad121b-s7qhk"] Oct 31 21:49:05.984951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:05.984918 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b1e5dbe-843c-48a0-9f0a-8722ce84aa11" path="/var/lib/kubelet/pods/4b1e5dbe-843c-48a0-9f0a-8722ce84aa11/volumes" Oct 31 21:49:08.971144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:08.969165 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b"] Oct 31 21:49:08.971652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:08.971573 4294 cpu_manager.go:414] "containerMap: removing container" podUID="34b3dab4-e9e1-42c6-baa9-1ad202b611eb" containerName="payload-pre-processing" Oct 31 21:49:08.971652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:08.971596 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b3dab4-e9e1-42c6-baa9-1ad202b611eb" containerName="payload-pre-processing" Oct 31 21:49:08.971999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:08.971968 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b3dab4-e9e1-42c6-baa9-1ad202b611eb" containerName="payload-pre-processing" Oct 31 21:49:08.977465 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:08.977444 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:08.977828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:08.977802 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b"] Oct 31 21:49:09.004227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.004203 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-44405d-dockercfg-74v5v\"" Oct 31 21:49:09.004457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.004438 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-44405d\"" Oct 31 21:49:09.158637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.158603 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvqj5\" (UniqueName: \"kubernetes.io/projected/583e6add-c5b4-42b6-a9af-5de6e8563a8c-kube-api-access-mvqj5\") pod \"payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:09.158800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.158644 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e6add-c5b4-42b6-a9af-5de6e8563a8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:09.259755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.259724 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvqj5\" (UniqueName: \"kubernetes.io/projected/583e6add-c5b4-42b6-a9af-5de6e8563a8c-kube-api-access-mvqj5\") pod \"payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:09.259948 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.259770 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e6add-c5b4-42b6-a9af-5de6e8563a8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:09.260596 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.260574 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e6add-c5b4-42b6-a9af-5de6e8563a8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:09.316386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.316353 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvqj5\" (UniqueName: \"kubernetes.io/projected/583e6add-c5b4-42b6-a9af-5de6e8563a8c-kube-api-access-mvqj5\") pod \"payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:09.591194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.591107 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:09.895123 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:49:09.895087 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod583e6add_c5b4_42b6_a9af_5de6e8563a8c.slice/crio-6cf5688e833567f040d0c82cab249d7abc30be71c6dd637a57dc80479a4bedab WatchSource:0}: Error finding container 6cf5688e833567f040d0c82cab249d7abc30be71c6dd637a57dc80479a4bedab: Status 404 returned error can't find the container with id 6cf5688e833567f040d0c82cab249d7abc30be71c6dd637a57dc80479a4bedab Oct 31 21:49:09.895728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:09.895599 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b"] Oct 31 21:49:10.584971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:10.584932 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" event={"ID":"583e6add-c5b4-42b6-a9af-5de6e8563a8c","Type":"ContainerStarted","Data":"3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764"} Oct 31 21:49:10.585448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:10.584995 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" event={"ID":"583e6add-c5b4-42b6-a9af-5de6e8563a8c","Type":"ContainerStarted","Data":"6cf5688e833567f040d0c82cab249d7abc30be71c6dd637a57dc80479a4bedab"} Oct 31 21:49:10.585448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:10.585120 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:10.629260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:10.629203 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" podStartSLOduration=2.629189159 podStartE2EDuration="2.629189159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:49:10.625372031 +0000 UTC m=+1990.750479261" watchObservedRunningTime="2026-10-31 21:49:10.629189159 +0000 UTC m=+1990.754296431" Oct 31 21:49:21.155368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.155330 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq"] Oct 31 21:49:21.161950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.161921 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.179862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.179835 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq"] Oct 31 21:49:21.184088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.184061 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:49:21.184088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.184066 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:49:21.263361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.263294 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.263560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.263441 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.263622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.263581 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6r7\" (UniqueName: \"kubernetes.io/projected/d32050fa-63f5-4305-81d2-54d854fafbcf-kube-api-access-6v6r7\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.365502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.365462 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.365707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.365556 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.365707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.365688 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6r7\" (UniqueName: \"kubernetes.io/projected/d32050fa-63f5-4305-81d2-54d854fafbcf-kube-api-access-6v6r7\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.366350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.366319 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.367975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.367951 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.389348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.389316 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6r7\" (UniqueName: \"kubernetes.io/projected/d32050fa-63f5-4305-81d2-54d854fafbcf-kube-api-access-6v6r7\") pod \"maas-api-revoke-sub-df92b58d-xc5qq\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.485828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.485799 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:21.598270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.596484 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 21:49:21.821776 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:49:21.821734 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32050fa_63f5_4305_81d2_54d854fafbcf.slice/crio-5e58bbc627b2a8cf318818a1ef3bcec1a56431e0d010120c4fb8eec1f9372327 WatchSource:0}: Error finding container 5e58bbc627b2a8cf318818a1ef3bcec1a56431e0d010120c4fb8eec1f9372327: Status 404 returned error can't find the container with id 5e58bbc627b2a8cf318818a1ef3bcec1a56431e0d010120c4fb8eec1f9372327 Oct 31 21:49:21.821926 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:21.821878 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq"] Oct 31 21:49:22.676094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:22.676054 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" event={"ID":"d32050fa-63f5-4305-81d2-54d854fafbcf","Type":"ContainerStarted","Data":"eb9199a404bfc365c3913e60e773980b63827aa5292d176e7a85a7911a0a98b5"} Oct 31 21:49:22.676563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:22.676119 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" event={"ID":"d32050fa-63f5-4305-81d2-54d854fafbcf","Type":"ContainerStarted","Data":"5e58bbc627b2a8cf318818a1ef3bcec1a56431e0d010120c4fb8eec1f9372327"} Oct 31 21:49:22.720401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:22.720335 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" podStartSLOduration=1.720318522 podStartE2EDuration="1.720318522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:49:22.715637965 +0000 UTC m=+2002.840745197" watchObservedRunningTime="2026-10-31 21:49:22.720318522 +0000 UTC m=+2002.845425750" Oct 31 21:49:23.244473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:23.244429 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:49:23.244679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:23.244494 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:49:23.687372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:23.687323 4294 generic.go:356] "Generic (PLEG): container finished" podID="d32050fa-63f5-4305-81d2-54d854fafbcf" containerID="eb9199a404bfc365c3913e60e773980b63827aa5292d176e7a85a7911a0a98b5" exitCode=0 Oct 31 21:49:23.687774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:23.687457 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" event={"ID":"d32050fa-63f5-4305-81d2-54d854fafbcf","Type":"ContainerDied","Data":"eb9199a404bfc365c3913e60e773980b63827aa5292d176e7a85a7911a0a98b5"} Oct 31 21:49:24.699691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:24.699649 4294 generic.go:356] "Generic (PLEG): container finished" podID="5d6dff6d-171c-4611-924d-034020433dd6" containerID="bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0" exitCode=0 Oct 31 21:49:24.701266 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:24.700214 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" event={"ID":"5d6dff6d-171c-4611-924d-034020433dd6","Type":"ContainerDied","Data":"bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0"} Oct 31 21:49:24.701266 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:24.700789 4294 scope.go:122] "RemoveContainer" containerID="bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0" Oct 31 21:49:24.703579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:24.703546 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:49:24.703699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:24.703614 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:49:24.703699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:24.703645 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:49:25.011538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.009474 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:25.071042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.071007 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-xks-ca\") pod \"d32050fa-63f5-4305-81d2-54d854fafbcf\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " Oct 31 21:49:25.071244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.071114 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-openshift-ca\") pod \"d32050fa-63f5-4305-81d2-54d854fafbcf\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " Oct 31 21:49:25.071244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.071162 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d32050fa-63f5-4305-81d2-54d854fafbcf-kube-api-access-6v6r7\" (UniqueName: \"kubernetes.io/projected/d32050fa-63f5-4305-81d2-54d854fafbcf-kube-api-access-6v6r7\") pod \"d32050fa-63f5-4305-81d2-54d854fafbcf\" (UID: \"d32050fa-63f5-4305-81d2-54d854fafbcf\") " Oct 31 21:49:25.071569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.071540 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-openshift-ca" pod "d32050fa-63f5-4305-81d2-54d854fafbcf" (UID: "d32050fa-63f5-4305-81d2-54d854fafbcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:49:25.071996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.071972 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:49:25.073566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.073539 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-xks-ca" pod "d32050fa-63f5-4305-81d2-54d854fafbcf" (UID: "d32050fa-63f5-4305-81d2-54d854fafbcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:49:25.073653 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.073612 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32050fa-63f5-4305-81d2-54d854fafbcf-kube-api-access-6v6r7" pod "d32050fa-63f5-4305-81d2-54d854fafbcf" (UID: "d32050fa-63f5-4305-81d2-54d854fafbcf"). InnerVolumeSpecName "kube-api-access-6v6r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:49:25.173335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.173277 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6r7\" (UniqueName: \"kubernetes.io/projected/d32050fa-63f5-4305-81d2-54d854fafbcf-kube-api-access-6v6r7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:49:25.173335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.173335 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d32050fa-63f5-4305-81d2-54d854fafbcf-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:49:25.709345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.709286 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" event={"ID":"5d6dff6d-171c-4611-924d-034020433dd6","Type":"ContainerStarted","Data":"597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa"} Oct 31 21:49:25.710755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.710732 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" Oct 31 21:49:25.710878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.710822 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq" event={"ID":"d32050fa-63f5-4305-81d2-54d854fafbcf","Type":"ContainerDied","Data":"5e58bbc627b2a8cf318818a1ef3bcec1a56431e0d010120c4fb8eec1f9372327"} Oct 31 21:49:25.710878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.710863 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e58bbc627b2a8cf318818a1ef3bcec1a56431e0d010120c4fb8eec1f9372327" Oct 31 21:49:25.889612 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.889579 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:49:25.889801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.889755 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 21:49:25.889875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:25.889808 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 21:49:26.896831 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:26.896787 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 21:49:26.897390 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:26.896867 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 21:49:27.895139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:27.893597 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 21:49:27.895139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:27.893671 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 21:49:28.892814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:28.892778 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 21:49:28.893438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:28.893407 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 21:49:29.896762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:29.896727 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 21:49:29.897261 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:29.896800 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 21:49:30.894719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:30.894677 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" start-of-body= Oct 31 21:49:30.895022 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:30.894987 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.162:15021/healthz/ready\": dial tcp 10.134.0.162:15021: connect: connection refused" Oct 31 21:49:31.323791 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:49:31.323757 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:49:31.770106 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.770067 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl"] Oct 31 21:49:31.771449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.771423 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d32050fa-63f5-4305-81d2-54d854fafbcf" containerName="revoke-keys" Oct 31 21:49:31.771913 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.771894 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d32050fa-63f5-4305-81d2-54d854fafbcf" containerName="revoke-keys" Oct 31 21:49:31.772235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.772215 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d32050fa-63f5-4305-81d2-54d854fafbcf" containerName="revoke-keys" Oct 31 21:49:31.789396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.789365 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.805015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.804982 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl"] Oct 31 21:49:31.816702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.810205 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b36e62\"/\"auth-test-model-b36e62-kserve-self-signed-certs\"" Oct 31 21:49:31.816702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.810297 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b36e62\"/\"kube-root-ca.crt\"" Oct 31 21:49:31.816702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.810945 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b36e62\"/\"openshift-service-ca.crt\"" Oct 31 21:49:31.818630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.818609 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b36e62\"/\"default-dockercfg-tk2hp\"" Oct 31 21:49:31.879605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.879569 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6be379-1483-4028-8935-4f727891e84f-tls-certs\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.879968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.879942 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c2mh\" (UniqueName: \"kubernetes.io/projected/4a6be379-1483-4028-8935-4f727891e84f-kube-api-access-9c2mh\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.880274 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.880246 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-dshm\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.880423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.880292 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-model-cache\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.880423 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.880380 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-tmp-dir\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.880540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.880428 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-home\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.896883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.896853 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:49:31.897824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.897799 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:49:31.899400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.899376 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 21:49:31.981836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.981798 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6be379-1483-4028-8935-4f727891e84f-tls-certs\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.982033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.981885 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c2mh\" (UniqueName: \"kubernetes.io/projected/4a6be379-1483-4028-8935-4f727891e84f-kube-api-access-9c2mh\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.982100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.982083 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-dshm\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.982156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.982119 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-model-cache\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.982211 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.982192 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-tmp-dir\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.982277 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.982242 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-home\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.982722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.982694 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-home\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.982927 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.982902 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-model-cache\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.985145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.985122 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-dshm\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:31.987708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:31.987677 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-tmp-dir\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:32.030559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:32.029061 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6be379-1483-4028-8935-4f727891e84f-tls-certs\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:32.055386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:32.047789 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c2mh\" (UniqueName: \"kubernetes.io/projected/4a6be379-1483-4028-8935-4f727891e84f-kube-api-access-9c2mh\") pod \"auth-test-model-b36e62-kserve-7cc785ff7-7jqkl\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:32.157660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:32.157627 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:32.642880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:32.642850 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl"] Oct 31 21:49:32.644136 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:49:32.644106 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a6be379_1483_4028_8935_4f727891e84f.slice/crio-ceb2890f720c4af25af60814b6c0d474904f4aef00535cb630a283c9ee697bfa WatchSource:0}: Error finding container ceb2890f720c4af25af60814b6c0d474904f4aef00535cb630a283c9ee697bfa: Status 404 returned error can't find the container with id ceb2890f720c4af25af60814b6c0d474904f4aef00535cb630a283c9ee697bfa Oct 31 21:49:32.803337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:32.803273 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" event={"ID":"4a6be379-1483-4028-8935-4f727891e84f","Type":"ContainerStarted","Data":"e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c"} Oct 31 21:49:32.803534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:32.803343 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" event={"ID":"4a6be379-1483-4028-8935-4f727891e84f","Type":"ContainerStarted","Data":"ceb2890f720c4af25af60814b6c0d474904f4aef00535cb630a283c9ee697bfa"} Oct 31 21:49:32.849049 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:32.848996 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" podStartSLOduration=1.848982335 podStartE2EDuration="1.848982335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:49:32.847708028 +0000 UTC m=+2012.972815259" watchObservedRunningTime="2026-10-31 21:49:32.848982335 +0000 UTC m=+2012.974089567" Oct 31 21:49:33.158344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:33.158166 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:42.157887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.157841 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:42.164039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.163800 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": read tcp 10.134.0.2:33028->10.134.0.163:15021: read: connection reset by peer" start-of-body= Oct 31 21:49:42.164039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.163876 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": read tcp 10.134.0.2:33028->10.134.0.163:15021: read: connection reset by peer" Oct 31 21:49:42.172828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.172802 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:42.911624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.911553 4294 generic.go:356] "Generic (PLEG): container finished" podID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerID="771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e" exitCode=0 Oct 31 21:49:42.911786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.911621 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" event={"ID":"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e","Type":"ContainerDied","Data":"771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e"} Oct 31 21:49:42.912106 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.912090 4294 scope.go:122] "RemoveContainer" containerID="771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e" Oct 31 21:49:42.913716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.913689 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:49:42.913802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.913760 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:49:42.913802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.913794 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:49:42.926244 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:42.926219 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:49:43.921016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:43.920979 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" event={"ID":"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e","Type":"ContainerStarted","Data":"8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b"} Oct 31 21:49:44.429733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:44.429698 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:49:44.429905 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:44.429889 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:49:44.429961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:44.429940 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 21:49:45.039787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:45.039755 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n"] Oct 31 21:49:45.051417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:45.051387 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56d321a4-x696n"] Oct 31 21:49:45.429688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:45.429599 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:49:45.429688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:45.429671 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 21:49:46.003065 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:46.003028 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f4d39d-d8fe-4ae6-9541-e3c9981213d6" path="/var/lib/kubelet/pods/c8f4d39d-d8fe-4ae6-9541-e3c9981213d6/volumes" Oct 31 21:49:46.429939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:46.429855 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:49:46.429939 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:46.429905 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 21:49:47.429554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:47.429511 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:49:47.429756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:47.429579 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 21:49:48.429284 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:48.429252 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 31 21:49:48.429691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:48.429333 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 31 21:49:49.433348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:49.433315 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:49:49.433847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:49.433821 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:49:49.434647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:49:49.434629 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 21:50:01.183865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.183836 4294 scope.go:122] "RemoveContainer" containerID="0c1c6c06183da4b53defa798d3c06d96b04988754c7d389a961f7d6229230b71" Oct 31 21:50:01.195587 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.195567 4294 scope.go:122] "RemoveContainer" containerID="57ffc4427bc3591d8c5f8120a3bb20feab8db303f71c403b7cc6ef44115fda13" Oct 31 21:50:01.313037 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:50:01.312994 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:50:01.329947 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.329917 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx"] Oct 31 21:50:01.336154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.336131 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.358083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.357696 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx"] Oct 31 21:50:01.436689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436592 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.436882 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436686 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.436882 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436765 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.436882 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436792 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.436882 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436830 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ck6z\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-kube-api-access-2ck6z\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.437158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436888 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66746da2-a008-48d8-a453-8349c639889c-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.437158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436919 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.437158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.436982 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.437158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.437043 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66746da2-a008-48d8-a453-8349c639889c-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.537975 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66746da2-a008-48d8-a453-8349c639889c-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538073 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538137 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538221 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538261 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538451 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538382 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ck6z\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-kube-api-access-2ck6z\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538467 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66746da2-a008-48d8-a453-8349c639889c-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538502 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538558 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538580 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538716 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-certs\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.538828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.538735 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-credential-socket\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.539086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.539061 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-data\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.539264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.539247 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66746da2-a008-48d8-a453-8349c639889c-istiod-ca-cert\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.540639 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.540615 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-envoy\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.540733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.540691 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66746da2-a008-48d8-a453-8349c639889c-istio-podinfo\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.562035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.562015 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ck6z\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-kube-api-access-2ck6z\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.562035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.562023 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-istio-token\") pod \"e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.663538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.663503 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:01.991858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.991831 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx"] Oct 31 21:50:01.992692 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:50:01.992664 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66746da2_a008_48d8_a453_8349c639889c.slice/crio-eae5e67fbfc7ae2d309051c979cd235ff427d113be35262a62a099f87ae0fc80 WatchSource:0}: Error finding container eae5e67fbfc7ae2d309051c979cd235ff427d113be35262a62a099f87ae0fc80: Status 404 returned error can't find the container with id eae5e67fbfc7ae2d309051c979cd235ff427d113be35262a62a099f87ae0fc80 Oct 31 21:50:01.995442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.995415 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:50:01.995518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.995485 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:50:01.995558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:01.995519 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:50:02.039373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:02.039337 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww"] Oct 31 21:50:02.059414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:02.052933 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba3a8207-vtbww"] Oct 31 21:50:02.096282 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:02.096248 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" event={"ID":"66746da2-a008-48d8-a453-8349c639889c","Type":"ContainerStarted","Data":"eae5e67fbfc7ae2d309051c979cd235ff427d113be35262a62a099f87ae0fc80"} Oct 31 21:50:03.118714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.118671 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" event={"ID":"66746da2-a008-48d8-a453-8349c639889c","Type":"ContainerStarted","Data":"3bc173ed94383494ba57a55aca4072e7aaf23317b7f548afb58b921aecd1b916"} Oct 31 21:50:03.172660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.172598 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podStartSLOduration=2.172577914 podStartE2EDuration="2.172577914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:50:03.170214042 +0000 UTC m=+2043.295321273" watchObservedRunningTime="2026-10-31 21:50:03.172577914 +0000 UTC m=+2043.297685161" Oct 31 21:50:03.664364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.664337 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:03.665998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.665966 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 21:50:03.666124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.666016 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 21:50:03.705324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.705035 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz"] Oct 31 21:50:03.712820 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.712791 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.718252 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.718208 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz"] Oct 31 21:50:03.728405 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.728379 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b5dab\"/\"openshift-service-ca.crt\"" Oct 31 21:50:03.730264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.730235 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b5dab\"/\"kube-root-ca.crt\"" Oct 31 21:50:03.740535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.740502 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b5dab\"/\"default-dockercfg-rqcsq\"" Oct 31 21:50:03.740535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.740520 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-6b5dab\"/\"auth-test-model-6b5dab-kserve-self-signed-certs\"" Oct 31 21:50:03.807214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.807181 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-home\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.807506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.807478 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-tmp-dir\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.807580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.807530 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-dshm\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.807640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.807619 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-model-cache\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.807697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.807680 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2624e7-297b-43b7-89c4-a46a67b5067c-tls-certs\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.807753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.807713 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8stn\" (UniqueName: \"kubernetes.io/projected/ef2624e7-297b-43b7-89c4-a46a67b5067c-kube-api-access-j8stn\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.909516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.909477 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-dshm\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.909685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.909533 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-model-cache\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.909685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.909583 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2624e7-297b-43b7-89c4-a46a67b5067c-tls-certs\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.909685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.909611 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8stn\" (UniqueName: \"kubernetes.io/projected/ef2624e7-297b-43b7-89c4-a46a67b5067c-kube-api-access-j8stn\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.909685 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.909659 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-home\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.909978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.909960 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-tmp-dir\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.910442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.910417 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-tmp-dir\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.911129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.911102 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-model-cache\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.911248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.911141 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-home\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.914168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.914124 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2624e7-297b-43b7-89c4-a46a67b5067c-tls-certs\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.916562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.916498 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-dshm\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:03.940381 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:03.940355 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8stn\" (UniqueName: \"kubernetes.io/projected/ef2624e7-297b-43b7-89c4-a46a67b5067c-kube-api-access-j8stn\") pod \"auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:04.004894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:04.004854 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82101bf8-5865-40b9-b9a3-fa58ded9fa45" path="/var/lib/kubelet/pods/82101bf8-5865-40b9-b9a3-fa58ded9fa45/volumes" Oct 31 21:50:04.035622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:04.035589 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:04.502355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:04.502324 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz"] Oct 31 21:50:04.502689 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:50:04.502519 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef2624e7_297b_43b7_89c4_a46a67b5067c.slice/crio-70f91a01c4b57ceb3367e343548f8399e3673033e29e437f5a10b876fc0403d8 WatchSource:0}: Error finding container 70f91a01c4b57ceb3367e343548f8399e3673033e29e437f5a10b876fc0403d8: Status 404 returned error can't find the container with id 70f91a01c4b57ceb3367e343548f8399e3673033e29e437f5a10b876fc0403d8 Oct 31 21:50:04.664807 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:04.664770 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 21:50:04.664975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:04.664828 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 21:50:05.202522 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:05.202483 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" event={"ID":"ef2624e7-297b-43b7-89c4-a46a67b5067c","Type":"ContainerStarted","Data":"51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb"} Oct 31 21:50:05.202699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:05.202530 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" event={"ID":"ef2624e7-297b-43b7-89c4-a46a67b5067c","Type":"ContainerStarted","Data":"70f91a01c4b57ceb3367e343548f8399e3673033e29e437f5a10b876fc0403d8"} Oct 31 21:50:05.245119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:05.245071 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" podStartSLOduration=2.245058189 podStartE2EDuration="2.245058189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:50:05.244373863 +0000 UTC m=+2045.369481093" watchObservedRunningTime="2026-10-31 21:50:05.245058189 +0000 UTC m=+2045.370165461" Oct 31 21:50:05.663800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:05.663768 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 21:50:05.664216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:05.663820 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 21:50:06.036249 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:06.036213 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:06.664166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:06.664128 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 21:50:06.664566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:06.664191 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 21:50:07.664253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:07.664220 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 21:50:07.664663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:07.664269 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 21:50:08.664905 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:08.664868 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 21:50:08.665320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:08.664931 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 21:50:09.668659 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:09.668630 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:09.669130 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:09.669042 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:09.669646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:09.669627 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:50:14.036388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:14.036341 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:14.051321 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:14.049554 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:14.306705 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:14.306624 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:50:15.039005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:15.038965 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99"] Oct 31 21:50:15.042733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:15.042703 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ee4c051-zjh99"] Oct 31 21:50:15.998851 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:15.998813 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43aa5a3a-cd3f-44ae-bd61-911a9c71a54a" path="/var/lib/kubelet/pods/43aa5a3a-cd3f-44ae-bd61-911a9c71a54a/volumes" Oct 31 21:50:21.704448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:21.704412 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:50:21.704824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:21.704477 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:50:23.365275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:23.365244 4294 generic.go:356] "Generic (PLEG): container finished" podID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerID="c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e" exitCode=0 Oct 31 21:50:23.365718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:23.365329 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" event={"ID":"1ef9228e-2e5c-4114-8187-423aa04fcae1","Type":"ContainerDied","Data":"c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e"} Oct 31 21:50:23.365718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:23.365690 4294 scope.go:122] "RemoveContainer" containerID="c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e" Oct 31 21:50:23.367198 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:23.367171 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:50:23.367280 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:23.367244 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:50:23.367342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:23.367278 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:50:24.391708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:24.391660 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" event={"ID":"1ef9228e-2e5c-4114-8187-423aa04fcae1","Type":"ContainerStarted","Data":"050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7"} Oct 31 21:50:25.160262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:25.160231 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:50:25.160453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:25.160403 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 21:50:25.160536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:25.160455 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 21:50:26.158991 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:26.158961 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 21:50:26.159429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:26.159017 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 21:50:27.159835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:27.159795 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 21:50:27.160264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:27.159858 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 21:50:28.049859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:28.049827 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b"] Oct 31 21:50:28.061136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:28.061113 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc17ebf4-lzh7b"] Oct 31 21:50:28.169578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:28.169396 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 21:50:28.170036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:28.169608 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 21:50:29.159737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:29.159704 4294 patch_prober.go:29] interesting pod/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 31 21:50:29.159912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:29.159763 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 31 21:50:29.989937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:29.989898 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1179a0fb-d792-4464-94fd-c894b3dd3b1c" path="/var/lib/kubelet/pods/1179a0fb-d792-4464-94fd-c894b3dd3b1c/volumes" Oct 31 21:50:30.163462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:30.163432 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:50:30.163886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:30.163871 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:50:30.164635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:30.164611 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 21:50:31.309905 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:50:31.309875 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:50:40.034405 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:40.034375 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5"] Oct 31 21:50:40.046256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:40.046231 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc9421a5-x9mn5"] Oct 31 21:50:41.034810 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:41.034779 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d"] Oct 31 21:50:41.046319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:41.046280 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c22e879-bdd7d"] Oct 31 21:50:41.985449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:41.985419 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231931ab-30d5-4b29-ab33-a9b5ffa14290" path="/var/lib/kubelet/pods/231931ab-30d5-4b29-ab33-a9b5ffa14290/volumes" Oct 31 21:50:41.985884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:41.985870 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec89a43-ef16-48e5-8d8c-01b8628b7e1f" path="/var/lib/kubelet/pods/7ec89a43-ef16-48e5-8d8c-01b8628b7e1f/volumes" Oct 31 21:50:51.034351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:51.034318 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz"] Oct 31 21:50:51.046593 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:51.046566 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-76549406-jcnmz"] Oct 31 21:50:51.985573 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:51.985542 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6690e865-0033-46a8-9c5c-93c1c3cb9ce5" path="/var/lib/kubelet/pods/6690e865-0033-46a8-9c5c-93c1c3cb9ce5/volumes" Oct 31 21:50:52.032924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:52.032897 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp"] Oct 31 21:50:52.044550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:52.044522 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6b20bd4-mczqp"] Oct 31 21:50:53.987120 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:53.987085 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14209b83-acb4-4b94-8166-8b295654577f" path="/var/lib/kubelet/pods/14209b83-acb4-4b94-8166-8b295654577f/volumes" Oct 31 21:50:55.905955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:55.905925 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw"] Oct 31 21:50:55.910318 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:55.910288 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:55.934436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:55.934414 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-44405d\"/\"openshift-service-ca.crt\"" Oct 31 21:50:55.945488 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:55.945467 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-44405d\"/\"kube-root-ca.crt\"" Oct 31 21:50:55.945591 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:55.945509 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-44405d\"/\"auth-test-model-44405d-kserve-self-signed-certs\"" Oct 31 21:50:55.945955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:55.945936 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-44405d\"/\"default-dockercfg-d8rhx\"" Oct 31 21:50:55.953009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:55.952980 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw"] Oct 31 21:50:56.022257 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.022220 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6lp\" (UniqueName: \"kubernetes.io/projected/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-kube-api-access-mt6lp\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.022477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.022265 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tls-certs\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.022477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.022389 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-home\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.022598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.022560 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-model-cache\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.022657 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.022605 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-dshm\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.022863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.022838 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tmp-dir\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.124661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.124617 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tmp-dir\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.124860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.124672 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6lp\" (UniqueName: \"kubernetes.io/projected/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-kube-api-access-mt6lp\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.124860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.124723 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tls-certs\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.124860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.124798 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-home\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.125033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.124918 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-model-cache\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.125033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.124958 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-dshm\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.125150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.125088 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tmp-dir\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.125400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.125370 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-home\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.125533 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.125408 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-model-cache\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.127207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.127185 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-dshm\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.127368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.127345 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tls-certs\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.148658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.148631 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6lp\" (UniqueName: \"kubernetes.io/projected/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-kube-api-access-mt6lp\") pod \"auth-test-model-44405d-kserve-548cd65697-hnxsw\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.227375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.224574 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:50:56.621287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.621260 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw"] Oct 31 21:50:56.622153 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:50:56.622127 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd73abf8e_f6b6_438f_abc9_2a8dbdc4754e.slice/crio-c0cb3443daab8b391e89510e3df585fed8946e81892776417bf2e41db6e5c39a WatchSource:0}: Error finding container c0cb3443daab8b391e89510e3df585fed8946e81892776417bf2e41db6e5c39a: Status 404 returned error can't find the container with id c0cb3443daab8b391e89510e3df585fed8946e81892776417bf2e41db6e5c39a Oct 31 21:50:56.685821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:56.685795 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" event={"ID":"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e","Type":"ContainerStarted","Data":"c0cb3443daab8b391e89510e3df585fed8946e81892776417bf2e41db6e5c39a"} Oct 31 21:50:57.696960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:57.696904 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" event={"ID":"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e","Type":"ContainerStarted","Data":"b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1"} Oct 31 21:50:57.743039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:57.742987 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" podStartSLOduration=2.7429741119999997 podStartE2EDuration="2.742974112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:50:57.739976841 +0000 UTC m=+2097.865084072" watchObservedRunningTime="2026-10-31 21:50:57.742974112 +0000 UTC m=+2097.868081376" Oct 31 21:50:58.228415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:50:58.228383 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:51:00.286986 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:00.286957 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:51:00.295929 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:00.295910 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:51:01.226038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:01.226009 4294 scope.go:122] "RemoveContainer" containerID="4f84f9e432feffca44960e81b23b16f23cfd4b09d2da7a32c36a25f8efc792d5" Oct 31 21:51:01.245170 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:01.245140 4294 scope.go:122] "RemoveContainer" containerID="1c1f0dec6ed4342254fd2736848b850ee296ed95a8b0890a7c83ceeadb01563c" Oct 31 21:51:01.257686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:01.257656 4294 scope.go:122] "RemoveContainer" containerID="4ab0a63056f98dfcaaa4fdcbfb947c2546bb126f3723576915e98927d05492da" Oct 31 21:51:01.271480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:01.271457 4294 scope.go:122] "RemoveContainer" containerID="1bd3546e070d6caa881ebd8a0335ce9944f544e36a32b3a07a512f367721eb88" Oct 31 21:51:01.282575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:01.282548 4294 scope.go:122] "RemoveContainer" containerID="e589d74323780e7ab03328d603d6e2f4668859a5badad5a7a6621b082a7b85d5" Oct 31 21:51:01.319423 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:51:01.319384 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:51:04.031627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:04.031588 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht"] Oct 31 21:51:04.045655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:04.045621 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-475a1f35-h75ht"] Oct 31 21:51:05.032393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:05.032360 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw"] Oct 31 21:51:05.044017 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:05.043990 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc9ca4ce-z7wdw"] Oct 31 21:51:06.006193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:06.006006 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4532111c-91ee-4579-99a0-e7d2c877ccc9" path="/var/lib/kubelet/pods/4532111c-91ee-4579-99a0-e7d2c877ccc9/volumes" Oct 31 21:51:06.006765 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:06.006742 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4dbd5c-bad9-4834-ad4e-d84f20e3651f" path="/var/lib/kubelet/pods/de4dbd5c-bad9-4834-ad4e-d84f20e3651f/volumes" Oct 31 21:51:06.228794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:06.228758 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:51:06.241428 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:06.241404 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:51:06.776573 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:06.776543 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:51:08.281989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.281953 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p"] Oct 31 21:51:08.288721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.288696 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.304321 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.303263 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p"] Oct 31 21:51:08.313319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.311922 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:51:08.313319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.312120 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:51:08.413383 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.413343 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752858c0-691e-4505-921f-414a13a6eb94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.413555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.413495 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw694\" (UniqueName: \"kubernetes.io/projected/752858c0-691e-4505-921f-414a13a6eb94-kube-api-access-zw694\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.413638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.413622 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752858c0-691e-4505-921f-414a13a6eb94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.515691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.515655 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752858c0-691e-4505-921f-414a13a6eb94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.515859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.515752 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw694\" (UniqueName: \"kubernetes.io/projected/752858c0-691e-4505-921f-414a13a6eb94-kube-api-access-zw694\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.515859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.515832 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752858c0-691e-4505-921f-414a13a6eb94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.516475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.516449 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752858c0-691e-4505-921f-414a13a6eb94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.517998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.517976 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752858c0-691e-4505-921f-414a13a6eb94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.539329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.539243 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw694\" (UniqueName: \"kubernetes.io/projected/752858c0-691e-4505-921f-414a13a6eb94-kube-api-access-zw694\") pod \"maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.605071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.605038 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:08.916012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:08.915979 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p"] Oct 31 21:51:08.916915 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:51:08.916890 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752858c0_691e_4505_921f_414a13a6eb94.slice/crio-fcfd469a5ffd562cb73e6aa173f6df3f59a799b347818f9d9471960b759cb324 WatchSource:0}: Error finding container fcfd469a5ffd562cb73e6aa173f6df3f59a799b347818f9d9471960b759cb324: Status 404 returned error can't find the container with id fcfd469a5ffd562cb73e6aa173f6df3f59a799b347818f9d9471960b759cb324 Oct 31 21:51:09.793536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:09.793506 4294 generic.go:356] "Generic (PLEG): container finished" podID="752858c0-691e-4505-921f-414a13a6eb94" containerID="0bfdfb99ab1f69ceeb745bb0e33017b1e5d9cb075c32bb025d7bcac3ea1c16ed" exitCode=0 Oct 31 21:51:09.793915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:09.793588 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" event={"ID":"752858c0-691e-4505-921f-414a13a6eb94","Type":"ContainerDied","Data":"0bfdfb99ab1f69ceeb745bb0e33017b1e5d9cb075c32bb025d7bcac3ea1c16ed"} Oct 31 21:51:09.793915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:09.793635 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" event={"ID":"752858c0-691e-4505-921f-414a13a6eb94","Type":"ContainerStarted","Data":"fcfd469a5ffd562cb73e6aa173f6df3f59a799b347818f9d9471960b759cb324"} Oct 31 21:51:10.952055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:10.952027 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:11.055722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.055641 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/752858c0-691e-4505-921f-414a13a6eb94-kube-api-access-zw694\" (UniqueName: \"kubernetes.io/projected/752858c0-691e-4505-921f-414a13a6eb94-kube-api-access-zw694\") pod \"752858c0-691e-4505-921f-414a13a6eb94\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " Oct 31 21:51:11.055722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.055689 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/752858c0-691e-4505-921f-414a13a6eb94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752858c0-691e-4505-921f-414a13a6eb94-maas-api-openshift-ca\") pod \"752858c0-691e-4505-921f-414a13a6eb94\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " Oct 31 21:51:11.055915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.055745 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/752858c0-691e-4505-921f-414a13a6eb94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752858c0-691e-4505-921f-414a13a6eb94-maas-api-xks-ca\") pod \"752858c0-691e-4505-921f-414a13a6eb94\" (UID: \"752858c0-691e-4505-921f-414a13a6eb94\") " Oct 31 21:51:11.056113 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.056082 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/752858c0-691e-4505-921f-414a13a6eb94-maas-api-openshift-ca" pod "752858c0-691e-4505-921f-414a13a6eb94" (UID: "752858c0-691e-4505-921f-414a13a6eb94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:51:11.057844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.057821 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/752858c0-691e-4505-921f-414a13a6eb94-maas-api-xks-ca" pod "752858c0-691e-4505-921f-414a13a6eb94" (UID: "752858c0-691e-4505-921f-414a13a6eb94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:51:11.057844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.057833 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/752858c0-691e-4505-921f-414a13a6eb94-kube-api-access-zw694" pod "752858c0-691e-4505-921f-414a13a6eb94" (UID: "752858c0-691e-4505-921f-414a13a6eb94"). InnerVolumeSpecName "kube-api-access-zw694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:51:11.157326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.157279 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw694\" (UniqueName: \"kubernetes.io/projected/752858c0-691e-4505-921f-414a13a6eb94-kube-api-access-zw694\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:11.157326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.157330 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/752858c0-691e-4505-921f-414a13a6eb94-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:11.157522 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.157346 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/752858c0-691e-4505-921f-414a13a6eb94-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:11.810460 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.810430 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" Oct 31 21:51:11.810644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.810479 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p" event={"ID":"752858c0-691e-4505-921f-414a13a6eb94","Type":"ContainerDied","Data":"fcfd469a5ffd562cb73e6aa173f6df3f59a799b347818f9d9471960b759cb324"} Oct 31 21:51:11.810644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:11.810522 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcfd469a5ffd562cb73e6aa173f6df3f59a799b347818f9d9471960b759cb324" Oct 31 21:51:17.234518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.234436 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8"] Oct 31 21:51:17.235230 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.235208 4294 cpu_manager.go:414] "containerMap: removing container" podUID="752858c0-691e-4505-921f-414a13a6eb94" containerName="revoke-keys" Oct 31 21:51:17.235347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.235230 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="752858c0-691e-4505-921f-414a13a6eb94" containerName="revoke-keys" Oct 31 21:51:17.235469 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.235455 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="752858c0-691e-4505-921f-414a13a6eb94" containerName="revoke-keys" Oct 31 21:51:17.239268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.239243 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.257509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.257480 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8"] Oct 31 21:51:17.265380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.265349 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:51:17.265535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.265457 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:51:17.305029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.304978 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.305220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.305200 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.305351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.305333 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvz4j\" (UniqueName: \"kubernetes.io/projected/892bf842-69b8-49d1-b64b-adb78810ba7f-kube-api-access-xvz4j\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.406399 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.406354 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvz4j\" (UniqueName: \"kubernetes.io/projected/892bf842-69b8-49d1-b64b-adb78810ba7f-kube-api-access-xvz4j\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.406577 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.406500 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.406721 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.406701 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.407377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.407354 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.408877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.408856 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.436659 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.436625 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvz4j\" (UniqueName: \"kubernetes.io/projected/892bf842-69b8-49d1-b64b-adb78810ba7f-kube-api-access-xvz4j\") pod \"maas-api-revoke-sub-e3134895-nn4c8\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.555812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.555738 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:17.878747 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:51:17.878709 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892bf842_69b8_49d1_b64b_adb78810ba7f.slice/crio-ef515848271bc6035158b33b756a999c1eb74585ba5a5117fbb81bc605688a50 WatchSource:0}: Error finding container ef515848271bc6035158b33b756a999c1eb74585ba5a5117fbb81bc605688a50: Status 404 returned error can't find the container with id ef515848271bc6035158b33b756a999c1eb74585ba5a5117fbb81bc605688a50 Oct 31 21:51:17.880067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.880038 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8"] Oct 31 21:51:17.898248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:17.898205 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" event={"ID":"892bf842-69b8-49d1-b64b-adb78810ba7f","Type":"ContainerStarted","Data":"ef515848271bc6035158b33b756a999c1eb74585ba5a5117fbb81bc605688a50"} Oct 31 21:51:18.909341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:18.909290 4294 generic.go:356] "Generic (PLEG): container finished" podID="892bf842-69b8-49d1-b64b-adb78810ba7f" containerID="34dde9afd4b25262d641dd71cc4360c170edb6fc8d7ed4bb7a1aecc99bf4c043" exitCode=0 Oct 31 21:51:18.909797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:18.909339 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" event={"ID":"892bf842-69b8-49d1-b64b-adb78810ba7f","Type":"ContainerDied","Data":"34dde9afd4b25262d641dd71cc4360c170edb6fc8d7ed4bb7a1aecc99bf4c043"} Oct 31 21:51:20.036287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.036257 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw"] Oct 31 21:51:20.043376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.043341 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01c6bcee-nrlvw"] Oct 31 21:51:20.125693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.125665 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:20.255220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.255188 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-xks-ca\") pod \"892bf842-69b8-49d1-b64b-adb78810ba7f\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " Oct 31 21:51:20.255414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.255384 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-openshift-ca\") pod \"892bf842-69b8-49d1-b64b-adb78810ba7f\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " Oct 31 21:51:20.255520 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.255504 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892bf842-69b8-49d1-b64b-adb78810ba7f-kube-api-access-xvz4j\" (UniqueName: \"kubernetes.io/projected/892bf842-69b8-49d1-b64b-adb78810ba7f-kube-api-access-xvz4j\") pod \"892bf842-69b8-49d1-b64b-adb78810ba7f\" (UID: \"892bf842-69b8-49d1-b64b-adb78810ba7f\") " Oct 31 21:51:20.255744 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.255720 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-openshift-ca" pod "892bf842-69b8-49d1-b64b-adb78810ba7f" (UID: "892bf842-69b8-49d1-b64b-adb78810ba7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:51:20.256238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.256213 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:20.257235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.257210 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-xks-ca" pod "892bf842-69b8-49d1-b64b-adb78810ba7f" (UID: "892bf842-69b8-49d1-b64b-adb78810ba7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:51:20.257750 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.257726 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892bf842-69b8-49d1-b64b-adb78810ba7f-kube-api-access-xvz4j" pod "892bf842-69b8-49d1-b64b-adb78810ba7f" (UID: "892bf842-69b8-49d1-b64b-adb78810ba7f"). InnerVolumeSpecName "kube-api-access-xvz4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:51:20.360508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.360469 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvz4j\" (UniqueName: \"kubernetes.io/projected/892bf842-69b8-49d1-b64b-adb78810ba7f-kube-api-access-xvz4j\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:20.360756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.360739 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/892bf842-69b8-49d1-b64b-adb78810ba7f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:20.839990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.839947 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p"] Oct 31 21:51:20.851338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.851286 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-efdee4-bjm6p"] Oct 31 21:51:20.928395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.928355 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" event={"ID":"892bf842-69b8-49d1-b64b-adb78810ba7f","Type":"ContainerDied","Data":"ef515848271bc6035158b33b756a999c1eb74585ba5a5117fbb81bc605688a50"} Oct 31 21:51:20.928561 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.928418 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef515848271bc6035158b33b756a999c1eb74585ba5a5117fbb81bc605688a50" Oct 31 21:51:20.928561 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:20.928419 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8" Oct 31 21:51:21.986827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:21.986796 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="752858c0-691e-4505-921f-414a13a6eb94" path="/var/lib/kubelet/pods/752858c0-691e-4505-921f-414a13a6eb94/volumes" Oct 31 21:51:21.987464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:21.987445 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece0dd47-f5af-4e69-ba8a-4c1985074337" path="/var/lib/kubelet/pods/ece0dd47-f5af-4e69-ba8a-4c1985074337/volumes" Oct 31 21:51:27.639765 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.639733 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t"] Oct 31 21:51:27.640386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.640371 4294 cpu_manager.go:414] "containerMap: removing container" podUID="892bf842-69b8-49d1-b64b-adb78810ba7f" containerName="revoke-keys" Oct 31 21:51:27.640386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.640387 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892bf842-69b8-49d1-b64b-adb78810ba7f" containerName="revoke-keys" Oct 31 21:51:27.640499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.640489 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="892bf842-69b8-49d1-b64b-adb78810ba7f" containerName="revoke-keys" Oct 31 21:51:27.644066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.644050 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.664484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.664460 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t"] Oct 31 21:51:27.670323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.670284 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:51:27.670464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.670209 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:51:27.766775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.766737 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.766930 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.766849 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ngw2\" (UniqueName: \"kubernetes.io/projected/7fbc5f98-973f-445a-ad11-84ca7703443d-kube-api-access-4ngw2\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.767000 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.766986 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.867952 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.867915 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.868148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.868036 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.868148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.868093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ngw2\" (UniqueName: \"kubernetes.io/projected/7fbc5f98-973f-445a-ad11-84ca7703443d-kube-api-access-4ngw2\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.868762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.868736 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.870329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.870286 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.891894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.891834 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ngw2\" (UniqueName: \"kubernetes.io/projected/7fbc5f98-973f-445a-ad11-84ca7703443d-kube-api-access-4ngw2\") pod \"maas-api-revoke-sub-dd8520a8-wzq6t\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:27.959502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:27.959479 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:28.275794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:28.275765 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t"] Oct 31 21:51:28.279019 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:51:28.278986 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbc5f98_973f_445a_ad11_84ca7703443d.slice/crio-4358787aa18e8d6239e90034f1c613d43ed6831423ab7454ce08889c60671045 WatchSource:0}: Error finding container 4358787aa18e8d6239e90034f1c613d43ed6831423ab7454ce08889c60671045: Status 404 returned error can't find the container with id 4358787aa18e8d6239e90034f1c613d43ed6831423ab7454ce08889c60671045 Oct 31 21:51:28.999692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:28.999660 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" event={"ID":"7fbc5f98-973f-445a-ad11-84ca7703443d","Type":"ContainerStarted","Data":"68c696f600ad9b081020e62b90568b15a2d2a6e890498fa255a1eab5d966bc19"} Oct 31 21:51:29.000139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:28.999712 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" event={"ID":"7fbc5f98-973f-445a-ad11-84ca7703443d","Type":"ContainerStarted","Data":"4358787aa18e8d6239e90034f1c613d43ed6831423ab7454ce08889c60671045"} Oct 31 21:51:29.043375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:29.043290 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" podStartSLOduration=2.043278391 podStartE2EDuration="2.043278391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:51:29.041476009 +0000 UTC m=+2129.166583240" watchObservedRunningTime="2026-10-31 21:51:29.043278391 +0000 UTC m=+2129.168385622" Oct 31 21:51:30.008104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:30.008069 4294 generic.go:356] "Generic (PLEG): container finished" podID="7fbc5f98-973f-445a-ad11-84ca7703443d" containerID="68c696f600ad9b081020e62b90568b15a2d2a6e890498fa255a1eab5d966bc19" exitCode=0 Oct 31 21:51:30.008553 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:30.008150 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" event={"ID":"7fbc5f98-973f-445a-ad11-84ca7703443d","Type":"ContainerDied","Data":"68c696f600ad9b081020e62b90568b15a2d2a6e890498fa255a1eab5d966bc19"} Oct 31 21:51:31.188213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.188182 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:31.224654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.224622 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fbc5f98-973f-445a-ad11-84ca7703443d-kube-api-access-4ngw2\" (UniqueName: \"kubernetes.io/projected/7fbc5f98-973f-445a-ad11-84ca7703443d-kube-api-access-4ngw2\") pod \"7fbc5f98-973f-445a-ad11-84ca7703443d\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " Oct 31 21:51:31.224654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.224658 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-openshift-ca\") pod \"7fbc5f98-973f-445a-ad11-84ca7703443d\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " Oct 31 21:51:31.224958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.224695 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-xks-ca\") pod \"7fbc5f98-973f-445a-ad11-84ca7703443d\" (UID: \"7fbc5f98-973f-445a-ad11-84ca7703443d\") " Oct 31 21:51:31.225104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.225080 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-openshift-ca" pod "7fbc5f98-973f-445a-ad11-84ca7703443d" (UID: "7fbc5f98-973f-445a-ad11-84ca7703443d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:51:31.225480 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.225457 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:31.226911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.226884 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbc5f98-973f-445a-ad11-84ca7703443d-kube-api-access-4ngw2" pod "7fbc5f98-973f-445a-ad11-84ca7703443d" (UID: "7fbc5f98-973f-445a-ad11-84ca7703443d"). InnerVolumeSpecName "kube-api-access-4ngw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:51:31.226911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.226887 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-xks-ca" pod "7fbc5f98-973f-445a-ad11-84ca7703443d" (UID: "7fbc5f98-973f-445a-ad11-84ca7703443d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:51:31.312656 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:51:31.312578 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:51:31.326618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.326590 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ngw2\" (UniqueName: \"kubernetes.io/projected/7fbc5f98-973f-445a-ad11-84ca7703443d-kube-api-access-4ngw2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:31.326618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:31.326619 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fbc5f98-973f-445a-ad11-84ca7703443d-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:51:32.035167 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:32.035125 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" event={"ID":"7fbc5f98-973f-445a-ad11-84ca7703443d","Type":"ContainerDied","Data":"4358787aa18e8d6239e90034f1c613d43ed6831423ab7454ce08889c60671045"} Oct 31 21:51:32.035332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:32.035174 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t" Oct 31 21:51:32.035332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:32.035179 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4358787aa18e8d6239e90034f1c613d43ed6831423ab7454ce08889c60671045" Oct 31 21:51:38.037194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:38.037155 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g"] Oct 31 21:51:38.049003 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:38.048974 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b10fe77b-97d2g"] Oct 31 21:51:39.985529 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:39.985496 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093719aa-559e-4a7d-a2c8-59386676956e" path="/var/lib/kubelet/pods/093719aa-559e-4a7d-a2c8-59386676956e/volumes" Oct 31 21:51:44.031800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:44.031764 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz"] Oct 31 21:51:44.043698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:44.043670 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42f3e9b9-v4tlz"] Oct 31 21:51:45.985259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:45.985219 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76e4c87b-7894-4980-9a06-7f7087adff5c" path="/var/lib/kubelet/pods/76e4c87b-7894-4980-9a06-7f7087adff5c/volumes" Oct 31 21:51:55.547725 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:55.547692 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd"] Oct 31 21:51:57.034481 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:57.034447 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65"] Oct 31 21:51:57.046102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:57.046077 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad7956-54s65"] Oct 31 21:51:57.985380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:51:57.985350 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ffeef8-93b1-4b63-922a-0f9b54b7b566" path="/var/lib/kubelet/pods/71ffeef8-93b1-4b63-922a-0f9b54b7b566/volumes" Oct 31 21:52:01.311602 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:01.311567 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:52:01.345378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.345351 4294 scope.go:122] "RemoveContainer" containerID="f408e23cd8041c422d11e97b30aea120162fa260195ddc56791cc104e9bc0df3" Oct 31 21:52:01.355482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.355457 4294 scope.go:122] "RemoveContainer" containerID="e8c36a4bb16599e21278a094c77038f4ba27298a81ad41123d50cf6424b5d23d" Oct 31 21:52:01.364524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.364507 4294 scope.go:122] "RemoveContainer" containerID="e3486b89192d9689869b61f50859921d36cd0d18ca6ed7c33834adc085bd7942" Oct 31 21:52:01.373458 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.373438 4294 scope.go:122] "RemoveContainer" containerID="31d93040c8ba8d74456d8050440e518d23b40dd24ec5aa9fa5aa6feb0b2a35cf" Oct 31 21:52:01.381989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.381969 4294 scope.go:122] "RemoveContainer" containerID="71faf1923788afd02fba3ac5cfcd1f0c7685fd510d744d18d6c8d1f6fcb80b9a" Oct 31 21:52:01.390571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.390552 4294 scope.go:122] "RemoveContainer" containerID="4d76bdad79d6b517eca45c0abfc09260164830ba96f2aabed16d942371184d53" Oct 31 21:52:01.399173 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.399153 4294 scope.go:122] "RemoveContainer" containerID="1c032eb93c7d4d145ca6b5e863543dd1cfccda10595a3744da36bd423ee0da86" Oct 31 21:52:01.407835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.407819 4294 scope.go:122] "RemoveContainer" containerID="929962e354f44eddcf12b7c220b4010ff9156ae5365d1f1c9d204673ee548570" Oct 31 21:52:01.416554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:01.416533 4294 scope.go:122] "RemoveContainer" containerID="c9d83274bb813c5f4e98c1e16fdfaa47b2e5f4aaecd83a0d2552ddac1122b52e" Oct 31 21:52:08.032852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:08.032821 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz"] Oct 31 21:52:08.044614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:08.044582 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d125d452-h2ffz"] Oct 31 21:52:09.985754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:09.985717 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3d6b131-fe5f-4e1e-9563-fcad55dca802" path="/var/lib/kubelet/pods/e3d6b131-fe5f-4e1e-9563-fcad55dca802/volumes" Oct 31 21:52:10.568269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.568213 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" podUID="a78f31ce-8e56-4e08-8319-a635ca291530" containerName="main" containerID="cri-o://d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616" gracePeriod=45 Oct 31 21:52:10.836747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.836684 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:52:10.893859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.893828 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-home\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-home\") pod \"a78f31ce-8e56-4e08-8319-a635ca291530\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " Oct 31 21:52:10.894038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.893992 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-model-cache\") pod \"a78f31ce-8e56-4e08-8319-a635ca291530\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " Oct 31 21:52:10.894150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.894101 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-home" pod "a78f31ce-8e56-4e08-8319-a635ca291530" (UID: "a78f31ce-8e56-4e08-8319-a635ca291530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:10.894150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.894135 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a78f31ce-8e56-4e08-8319-a635ca291530-kube-api-access-drd5w\" (UniqueName: \"kubernetes.io/projected/a78f31ce-8e56-4e08-8319-a635ca291530-kube-api-access-drd5w\") pod \"a78f31ce-8e56-4e08-8319-a635ca291530\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " Oct 31 21:52:10.894268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.894220 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-dshm\") pod \"a78f31ce-8e56-4e08-8319-a635ca291530\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " Oct 31 21:52:10.894268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.894260 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-tmp-dir\") pod \"a78f31ce-8e56-4e08-8319-a635ca291530\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " Oct 31 21:52:10.894387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.894334 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-model-cache" pod "a78f31ce-8e56-4e08-8319-a635ca291530" (UID: "a78f31ce-8e56-4e08-8319-a635ca291530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:10.894387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.894346 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a78f31ce-8e56-4e08-8319-a635ca291530-tls-certs\" (UniqueName: \"kubernetes.io/secret/a78f31ce-8e56-4e08-8319-a635ca291530-tls-certs\") pod \"a78f31ce-8e56-4e08-8319-a635ca291530\" (UID: \"a78f31ce-8e56-4e08-8319-a635ca291530\") " Oct 31 21:52:10.896417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.894983 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:10.896417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.895008 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:10.896417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.896371 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-tmp-dir" pod "a78f31ce-8e56-4e08-8319-a635ca291530" (UID: "a78f31ce-8e56-4e08-8319-a635ca291530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:10.897317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.897246 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-dshm" pod "a78f31ce-8e56-4e08-8319-a635ca291530" (UID: "a78f31ce-8e56-4e08-8319-a635ca291530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:52:10.898015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.897579 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a78f31ce-8e56-4e08-8319-a635ca291530-tls-certs" pod "a78f31ce-8e56-4e08-8319-a635ca291530" (UID: "a78f31ce-8e56-4e08-8319-a635ca291530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:52:10.898015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.897901 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a78f31ce-8e56-4e08-8319-a635ca291530-kube-api-access-drd5w" pod "a78f31ce-8e56-4e08-8319-a635ca291530" (UID: "a78f31ce-8e56-4e08-8319-a635ca291530"). InnerVolumeSpecName "kube-api-access-drd5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:52:10.996715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.996681 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drd5w\" (UniqueName: \"kubernetes.io/projected/a78f31ce-8e56-4e08-8319-a635ca291530-kube-api-access-drd5w\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:10.996715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.996709 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:10.996715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.996718 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a78f31ce-8e56-4e08-8319-a635ca291530-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:10.997147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:10.996729 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a78f31ce-8e56-4e08-8319-a635ca291530-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:11.356587 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.356556 4294 generic.go:356] "Generic (PLEG): container finished" podID="a78f31ce-8e56-4e08-8319-a635ca291530" containerID="d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616" exitCode=0 Oct 31 21:52:11.356780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.356600 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" Oct 31 21:52:11.356780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.356636 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" event={"ID":"a78f31ce-8e56-4e08-8319-a635ca291530","Type":"ContainerDied","Data":"d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616"} Oct 31 21:52:11.356780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.356679 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd" event={"ID":"a78f31ce-8e56-4e08-8319-a635ca291530","Type":"ContainerDied","Data":"454ad0edb11bcad351419c37d34c84f74cda5abf1109107c38ed5b5b11e39871"} Oct 31 21:52:11.356780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.356698 4294 scope.go:122] "RemoveContainer" containerID="d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616" Oct 31 21:52:11.367700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.367685 4294 scope.go:122] "RemoveContainer" containerID="d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616" Oct 31 21:52:11.367966 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:11.367947 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616\": container with ID starting with d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616 not found: ID does not exist" containerID="d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616" Oct 31 21:52:11.368032 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.367974 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616"} err="failed to get container status \"d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616\": rpc error: code = NotFound desc = could not find container \"d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616\": container with ID starting with d6835cb0f9fe4b1474ef6dcabc562e0379dc4d26720fb161d2f63d7866748616 not found: ID does not exist" Oct 31 21:52:11.409811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.409781 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd"] Oct 31 21:52:11.421957 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.421935 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-f1e791/auth-test-model-f1e791-kserve-7f8855758f-sccfd"] Oct 31 21:52:11.985944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:11.985913 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a78f31ce-8e56-4e08-8319-a635ca291530" path="/var/lib/kubelet/pods/a78f31ce-8e56-4e08-8319-a635ca291530/volumes" Oct 31 21:52:14.267798 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.267758 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7552-30c4ba"] Oct 31 21:52:14.268654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.268628 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7fbc5f98-973f-445a-ad11-84ca7703443d" containerName="revoke-keys" Oct 31 21:52:14.268717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.268658 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fbc5f98-973f-445a-ad11-84ca7703443d" containerName="revoke-keys" Oct 31 21:52:14.268717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.268683 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a78f31ce-8e56-4e08-8319-a635ca291530" containerName="main" Oct 31 21:52:14.268717 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.268690 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a78f31ce-8e56-4e08-8319-a635ca291530" containerName="main" Oct 31 21:52:14.268894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.268880 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fbc5f98-973f-445a-ad11-84ca7703443d" containerName="revoke-keys" Oct 31 21:52:14.268950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.268899 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a78f31ce-8e56-4e08-8319-a635ca291530" containerName="main" Oct 31 21:52:14.272973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.272947 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:14.286793 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.286764 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-30c4ba"] Oct 31 21:52:14.294654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.294612 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dsnkq\"" Oct 31 21:52:14.354962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.354920 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f747g\" (UniqueName: \"kubernetes.io/projected/ae3ed4e7-441c-45d4-8141-efbfc6af83d8-kube-api-access-f747g\") pod \"test-curl-7552-30c4ba\" (UID: \"ae3ed4e7-441c-45d4-8141-efbfc6af83d8\") " pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:14.456346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.456282 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f747g\" (UniqueName: \"kubernetes.io/projected/ae3ed4e7-441c-45d4-8141-efbfc6af83d8-kube-api-access-f747g\") pod \"test-curl-7552-30c4ba\" (UID: \"ae3ed4e7-441c-45d4-8141-efbfc6af83d8\") " pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:14.479950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.479917 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f747g\" (UniqueName: \"kubernetes.io/projected/ae3ed4e7-441c-45d4-8141-efbfc6af83d8-kube-api-access-f747g\") pod \"test-curl-7552-30c4ba\" (UID: \"ae3ed4e7-441c-45d4-8141-efbfc6af83d8\") " pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:14.590214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.590128 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:14.895782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:14.895744 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-30c4ba"] Oct 31 21:52:14.896576 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:52:14.896552 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae3ed4e7_441c_45d4_8141_efbfc6af83d8.slice/crio-9ec9bb5f2dc14288f63cb3196d58bdeefc6b6a9c5b674491b894a83aae237b9e WatchSource:0}: Error finding container 9ec9bb5f2dc14288f63cb3196d58bdeefc6b6a9c5b674491b894a83aae237b9e: Status 404 returned error can't find the container with id 9ec9bb5f2dc14288f63cb3196d58bdeefc6b6a9c5b674491b894a83aae237b9e Oct 31 21:52:15.043958 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:15.043925 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz"] Oct 31 21:52:15.047173 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:15.047149 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c05638a-28lgz"] Oct 31 21:52:15.390424 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:15.390392 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-30c4ba" event={"ID":"ae3ed4e7-441c-45d4-8141-efbfc6af83d8","Type":"ContainerStarted","Data":"1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3"} Oct 31 21:52:15.390927 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:15.390435 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-30c4ba" event={"ID":"ae3ed4e7-441c-45d4-8141-efbfc6af83d8","Type":"ContainerStarted","Data":"9ec9bb5f2dc14288f63cb3196d58bdeefc6b6a9c5b674491b894a83aae237b9e"} Oct 31 21:52:15.430946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:15.430896 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7552-30c4ba" podStartSLOduration=1.430884166 podStartE2EDuration="1.430884166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:52:15.429062842 +0000 UTC m=+2175.554170072" watchObservedRunningTime="2026-10-31 21:52:15.430884166 +0000 UTC m=+2175.555991397" Oct 31 21:52:15.985340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:15.985269 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17821818-4501-419d-898f-636cb0d56577" path="/var/lib/kubelet/pods/17821818-4501-419d-898f-636cb0d56577/volumes" Oct 31 21:52:16.441584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.441493 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7552-30c4ba"] Oct 31 21:52:16.447495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.447468 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7552-30c4ba"] Oct 31 21:52:16.696109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.696013 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7552-8bb9b0"] Oct 31 21:52:16.697623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.697593 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ae3ed4e7-441c-45d4-8141-efbfc6af83d8" containerName="test-curl-7552-30c4ba" Oct 31 21:52:16.697623 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.697622 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3ed4e7-441c-45d4-8141-efbfc6af83d8" containerName="test-curl-7552-30c4ba" Oct 31 21:52:16.697899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.697882 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3ed4e7-441c-45d4-8141-efbfc6af83d8" containerName="test-curl-7552-30c4ba" Oct 31 21:52:16.702448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.702425 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:16.715267 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.715242 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-8bb9b0"] Oct 31 21:52:16.902602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:16.902563 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cts4\" (UniqueName: \"kubernetes.io/projected/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b-kube-api-access-7cts4\") pod \"test-curl-7552-8bb9b0\" (UID: \"f6c78a10-58b7-4238-9f2b-bdc1d628ac7b\") " pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:17.003710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:17.003677 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cts4\" (UniqueName: \"kubernetes.io/projected/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b-kube-api-access-7cts4\") pod \"test-curl-7552-8bb9b0\" (UID: \"f6c78a10-58b7-4238-9f2b-bdc1d628ac7b\") " pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:17.029477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:17.029444 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cts4\" (UniqueName: \"kubernetes.io/projected/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b-kube-api-access-7cts4\") pod \"test-curl-7552-8bb9b0\" (UID: \"f6c78a10-58b7-4238-9f2b-bdc1d628ac7b\") " pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:17.316714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:17.316639 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:17.436956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:17.436053 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7552-30c4ba" podUID="ae3ed4e7-441c-45d4-8141-efbfc6af83d8" containerName="test-curl-7552-30c4ba" containerID="cri-o://1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3" gracePeriod=2 Oct 31 21:52:17.449635 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:17.449591 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7552-30c4ba\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="ae3ed4e7-441c-45d4-8141-efbfc6af83d8" pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:17.623196 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:17.623158 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-8bb9b0"] Oct 31 21:52:17.624590 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:52:17.624558 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6c78a10_58b7_4238_9f2b_bdc1d628ac7b.slice/crio-bced846ef1603e6143ae9ed00665b69658bb47f47dc664d3e4a515cb74a9f16a WatchSource:0}: Error finding container bced846ef1603e6143ae9ed00665b69658bb47f47dc664d3e4a515cb74a9f16a: Status 404 returned error can't find the container with id bced846ef1603e6143ae9ed00665b69658bb47f47dc664d3e4a515cb74a9f16a Oct 31 21:52:18.446400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:18.446354 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-8bb9b0" event={"ID":"f6c78a10-58b7-4238-9f2b-bdc1d628ac7b","Type":"ContainerStarted","Data":"a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225"} Oct 31 21:52:18.446601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:18.446428 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-8bb9b0" event={"ID":"f6c78a10-58b7-4238-9f2b-bdc1d628ac7b","Type":"ContainerStarted","Data":"bced846ef1603e6143ae9ed00665b69658bb47f47dc664d3e4a515cb74a9f16a"} Oct 31 21:52:18.527321 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:18.526798 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7552-8bb9b0" podStartSLOduration=2.526776506 podStartE2EDuration="2.526776506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:52:18.507944548 +0000 UTC m=+2178.633051781" watchObservedRunningTime="2026-10-31 21:52:18.526776506 +0000 UTC m=+2178.651883738" Oct 31 21:52:19.409021 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.408981 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7552-8bb9b0"] Oct 31 21:52:19.416454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.416418 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7552-8bb9b0"] Oct 31 21:52:19.561592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.561559 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7552-4ccea5"] Oct 31 21:52:19.562180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.562165 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" containerName="test-curl-7552-8bb9b0" Oct 31 21:52:19.562232 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.562182 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" containerName="test-curl-7552-8bb9b0" Oct 31 21:52:19.562337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.562297 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" containerName="test-curl-7552-8bb9b0" Oct 31 21:52:19.565838 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.565814 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:19.583455 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.583424 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-4ccea5"] Oct 31 21:52:19.663869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.663798 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85856\" (UniqueName: \"kubernetes.io/projected/c2a398ee-a7cc-4921-8142-948bc6809846-kube-api-access-85856\") pod \"test-curl-7552-4ccea5\" (UID: \"c2a398ee-a7cc-4921-8142-948bc6809846\") " pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:19.673652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.673630 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:19.764973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.764943 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3ed4e7-441c-45d4-8141-efbfc6af83d8-kube-api-access-f747g\" (UniqueName: \"kubernetes.io/projected/ae3ed4e7-441c-45d4-8141-efbfc6af83d8-kube-api-access-f747g\") pod \"ae3ed4e7-441c-45d4-8141-efbfc6af83d8\" (UID: \"ae3ed4e7-441c-45d4-8141-efbfc6af83d8\") " Oct 31 21:52:19.765259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.765240 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85856\" (UniqueName: \"kubernetes.io/projected/c2a398ee-a7cc-4921-8142-948bc6809846-kube-api-access-85856\") pod \"test-curl-7552-4ccea5\" (UID: \"c2a398ee-a7cc-4921-8142-948bc6809846\") " pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:19.767237 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.767210 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3ed4e7-441c-45d4-8141-efbfc6af83d8-kube-api-access-f747g" pod "ae3ed4e7-441c-45d4-8141-efbfc6af83d8" (UID: "ae3ed4e7-441c-45d4-8141-efbfc6af83d8"). InnerVolumeSpecName "kube-api-access-f747g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:52:19.788899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.788877 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85856\" (UniqueName: \"kubernetes.io/projected/c2a398ee-a7cc-4921-8142-948bc6809846-kube-api-access-85856\") pod \"test-curl-7552-4ccea5\" (UID: \"c2a398ee-a7cc-4921-8142-948bc6809846\") " pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:19.867097 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.867064 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f747g\" (UniqueName: \"kubernetes.io/projected/ae3ed4e7-441c-45d4-8141-efbfc6af83d8-kube-api-access-f747g\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:19.880466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.880445 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:19.991144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:19.991115 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3ed4e7-441c-45d4-8141-efbfc6af83d8" path="/var/lib/kubelet/pods/ae3ed4e7-441c-45d4-8141-efbfc6af83d8/volumes" Oct 31 21:52:20.188687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.188599 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-4ccea5"] Oct 31 21:52:20.189896 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:52:20.189871 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2a398ee_a7cc_4921_8142_948bc6809846.slice/crio-f1661b9914ff95b33372b6e68709dccd347d410905fe284822a80cb224c8d658 WatchSource:0}: Error finding container f1661b9914ff95b33372b6e68709dccd347d410905fe284822a80cb224c8d658: Status 404 returned error can't find the container with id f1661b9914ff95b33372b6e68709dccd347d410905fe284822a80cb224c8d658 Oct 31 21:52:20.467087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.467000 4294 generic.go:356] "Generic (PLEG): container finished" podID="ae3ed4e7-441c-45d4-8141-efbfc6af83d8" containerID="1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3" exitCode=137 Oct 31 21:52:20.467087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.467059 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-30c4ba" Oct 31 21:52:20.467087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.467080 4294 scope.go:122] "RemoveContainer" containerID="1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3" Oct 31 21:52:20.469297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.469030 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-4ccea5" event={"ID":"c2a398ee-a7cc-4921-8142-948bc6809846","Type":"ContainerStarted","Data":"24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee"} Oct 31 21:52:20.469297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.469101 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-4ccea5" event={"ID":"c2a398ee-a7cc-4921-8142-948bc6809846","Type":"ContainerStarted","Data":"f1661b9914ff95b33372b6e68709dccd347d410905fe284822a80cb224c8d658"} Oct 31 21:52:20.469297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.469133 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7552-8bb9b0" podUID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" containerName="test-curl-7552-8bb9b0" containerID="cri-o://a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225" gracePeriod=2 Oct 31 21:52:20.479620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.479602 4294 scope.go:122] "RemoveContainer" containerID="1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3" Oct 31 21:52:20.479876 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:20.479858 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3\": container with ID starting with 1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3 not found: ID does not exist" containerID="1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3" Oct 31 21:52:20.479943 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.479888 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3"} err="failed to get container status \"1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3\": rpc error: code = NotFound desc = could not find container \"1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3\": container with ID starting with 1c06bc34d80779f5d5cde6683104fb35c4d4db92139c4fe9fb066520bcbb06e3 not found: ID does not exist" Oct 31 21:52:20.511849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:20.511807 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7552-4ccea5" podStartSLOduration=1.511795314 podStartE2EDuration="1.511795314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:52:20.50776169 +0000 UTC m=+2180.632868920" watchObservedRunningTime="2026-10-31 21:52:20.511795314 +0000 UTC m=+2180.636902543" Oct 31 21:52:21.411190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.410934 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7552-4ccea5"] Oct 31 21:52:21.415552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.415527 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7552-4ccea5"] Oct 31 21:52:21.554443 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.554411 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7552-c259b0"] Oct 31 21:52:21.554984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.554971 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c2a398ee-a7cc-4921-8142-948bc6809846" containerName="test-curl-7552-4ccea5" Oct 31 21:52:21.555028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.554985 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a398ee-a7cc-4921-8142-948bc6809846" containerName="test-curl-7552-4ccea5" Oct 31 21:52:21.555115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.555099 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a398ee-a7cc-4921-8142-948bc6809846" containerName="test-curl-7552-4ccea5" Oct 31 21:52:21.559923 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.559904 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:21.572644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.572620 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-c259b0"] Oct 31 21:52:21.592507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.592477 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm9qz\" (UniqueName: \"kubernetes.io/projected/b2352bf9-6de7-42af-a6b7-cab92c8c915e-kube-api-access-tm9qz\") pod \"test-curl-7552-c259b0\" (UID: \"b2352bf9-6de7-42af-a6b7-cab92c8c915e\") " pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:21.693499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.693418 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm9qz\" (UniqueName: \"kubernetes.io/projected/b2352bf9-6de7-42af-a6b7-cab92c8c915e-kube-api-access-tm9qz\") pod \"test-curl-7552-c259b0\" (UID: \"b2352bf9-6de7-42af-a6b7-cab92c8c915e\") " pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:21.717229 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.717199 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm9qz\" (UniqueName: \"kubernetes.io/projected/b2352bf9-6de7-42af-a6b7-cab92c8c915e-kube-api-access-tm9qz\") pod \"test-curl-7552-c259b0\" (UID: \"b2352bf9-6de7-42af-a6b7-cab92c8c915e\") " pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:21.874188 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:21.874155 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:22.036948 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.036916 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4"] Oct 31 21:52:22.048944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.048910 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40434452-42bb4"] Oct 31 21:52:22.176859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.176832 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-c259b0"] Oct 31 21:52:22.178086 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:52:22.178060 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2352bf9_6de7_42af_a6b7_cab92c8c915e.slice/crio-fab02353b4bf8302816b4dd065e9567dfea5ab24b2af52e0318b0c42a004c040 WatchSource:0}: Error finding container fab02353b4bf8302816b4dd065e9567dfea5ab24b2af52e0318b0c42a004c040: Status 404 returned error can't find the container with id fab02353b4bf8302816b4dd065e9567dfea5ab24b2af52e0318b0c42a004c040 Oct 31 21:52:22.487936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.487907 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-c259b0" event={"ID":"b2352bf9-6de7-42af-a6b7-cab92c8c915e","Type":"ContainerStarted","Data":"88ec132e0f27f071600995ec44550565d447dc75fd3b044dca4ef1f5bdf97dee"} Oct 31 21:52:22.488256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.487956 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-c259b0" event={"ID":"b2352bf9-6de7-42af-a6b7-cab92c8c915e","Type":"ContainerStarted","Data":"fab02353b4bf8302816b4dd065e9567dfea5ab24b2af52e0318b0c42a004c040"} Oct 31 21:52:22.488256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.488111 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7552-4ccea5" podUID="c2a398ee-a7cc-4921-8142-948bc6809846" containerName="test-curl-7552-4ccea5" containerID="cri-o://24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee" gracePeriod=2 Oct 31 21:52:22.536924 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.536859 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7552-c259b0" podStartSLOduration=1.536839361 podStartE2EDuration="1.536839361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:52:22.525461423 +0000 UTC m=+2182.650568654" watchObservedRunningTime="2026-10-31 21:52:22.536839361 +0000 UTC m=+2182.661946592" Oct 31 21:52:22.647527 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.647499 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:22.707472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.707439 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b-kube-api-access-7cts4\" (UniqueName: \"kubernetes.io/projected/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b-kube-api-access-7cts4\") pod \"f6c78a10-58b7-4238-9f2b-bdc1d628ac7b\" (UID: \"f6c78a10-58b7-4238-9f2b-bdc1d628ac7b\") " Oct 31 21:52:22.710400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.710365 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b-kube-api-access-7cts4" pod "f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" (UID: "f6c78a10-58b7-4238-9f2b-bdc1d628ac7b"). InnerVolumeSpecName "kube-api-access-7cts4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:52:22.809400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:22.809371 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cts4\" (UniqueName: \"kubernetes.io/projected/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b-kube-api-access-7cts4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:23.373655 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.373621 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7552-c259b0"] Oct 31 21:52:23.376953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.376923 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7552-c259b0"] Oct 31 21:52:23.496494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.496461 4294 generic.go:356] "Generic (PLEG): container finished" podID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" containerID="a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225" exitCode=137 Oct 31 21:52:23.496906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.496510 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:23.496906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.496559 4294 scope.go:122] "RemoveContainer" containerID="a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225" Oct 31 21:52:23.506479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.506463 4294 scope.go:122] "RemoveContainer" containerID="a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225" Oct 31 21:52:23.506746 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:23.506720 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225\": container with ID starting with a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225 not found: ID does not exist" containerID="a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225" Oct 31 21:52:23.506833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.506750 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225"} err="failed to get container status \"a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225\": rpc error: code = NotFound desc = could not find container \"a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225\": container with ID starting with a80e4fd2c0d4cf8f44a9814b6e1cefb5bdea2900c16810a4c929ac19c7849225 not found: ID does not exist" Oct 31 21:52:23.520555 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:23.520531 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7552-8bb9b0\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:23.540992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.540970 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7552-fd65ec"] Oct 31 21:52:23.541562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.541547 4294 cpu_manager.go:414] "containerMap: removing container" podUID="b2352bf9-6de7-42af-a6b7-cab92c8c915e" containerName="test-curl-7552-c259b0" Oct 31 21:52:23.541615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.541564 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2352bf9-6de7-42af-a6b7-cab92c8c915e" containerName="test-curl-7552-c259b0" Oct 31 21:52:23.541722 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.541712 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2352bf9-6de7-42af-a6b7-cab92c8c915e" containerName="test-curl-7552-c259b0" Oct 31 21:52:23.546384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.546369 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:23.564022 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:23.563997 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7552-8bb9b0\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" pod="opendatahub/test-curl-7552-8bb9b0" Oct 31 21:52:23.576647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.576627 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-fd65ec"] Oct 31 21:52:23.621097 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.621066 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr28\" (UniqueName: \"kubernetes.io/projected/ee38fa5a-d473-4a99-8a14-4050b0224e3e-kube-api-access-cgr28\") pod \"test-curl-7552-fd65ec\" (UID: \"ee38fa5a-d473-4a99-8a14-4050b0224e3e\") " pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:23.723101 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.723023 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr28\" (UniqueName: \"kubernetes.io/projected/ee38fa5a-d473-4a99-8a14-4050b0224e3e-kube-api-access-cgr28\") pod \"test-curl-7552-fd65ec\" (UID: \"ee38fa5a-d473-4a99-8a14-4050b0224e3e\") " pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:23.746854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.746826 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr28\" (UniqueName: \"kubernetes.io/projected/ee38fa5a-d473-4a99-8a14-4050b0224e3e-kube-api-access-cgr28\") pod \"test-curl-7552-fd65ec\" (UID: \"ee38fa5a-d473-4a99-8a14-4050b0224e3e\") " pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:23.858577 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.858547 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:23.986714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.986682 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10e472a3-3c0f-4567-b56d-4a74047ceb02" path="/var/lib/kubelet/pods/10e472a3-3c0f-4567-b56d-4a74047ceb02/volumes" Oct 31 21:52:23.987175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:23.987150 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c78a10-58b7-4238-9f2b-bdc1d628ac7b" path="/var/lib/kubelet/pods/f6c78a10-58b7-4238-9f2b-bdc1d628ac7b/volumes" Oct 31 21:52:24.162769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.162740 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7552-fd65ec"] Oct 31 21:52:24.163470 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:52:24.163447 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee38fa5a_d473_4a99_8a14_4050b0224e3e.slice/crio-689f01be74d50c7a08130efb657d0c02b00a6ffd694dec4e5b0e790b5024d4b0 WatchSource:0}: Error finding container 689f01be74d50c7a08130efb657d0c02b00a6ffd694dec4e5b0e790b5024d4b0: Status 404 returned error can't find the container with id 689f01be74d50c7a08130efb657d0c02b00a6ffd694dec4e5b0e790b5024d4b0 Oct 31 21:52:24.527470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.527429 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-fd65ec" event={"ID":"ee38fa5a-d473-4a99-8a14-4050b0224e3e","Type":"ContainerStarted","Data":"07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b"} Oct 31 21:52:24.527977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.527489 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7552-fd65ec" event={"ID":"ee38fa5a-d473-4a99-8a14-4050b0224e3e","Type":"ContainerStarted","Data":"689f01be74d50c7a08130efb657d0c02b00a6ffd694dec4e5b0e790b5024d4b0"} Oct 31 21:52:24.527977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.527428 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7552-c259b0" podUID="b2352bf9-6de7-42af-a6b7-cab92c8c915e" containerName="test-curl-7552-c259b0" containerID="cri-o://88ec132e0f27f071600995ec44550565d447dc75fd3b044dca4ef1f5bdf97dee" gracePeriod=2 Oct 31 21:52:24.567812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.567764 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7552-fd65ec" podStartSLOduration=1.5677497900000001 podStartE2EDuration="1.56774979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:52:24.565751099 +0000 UTC m=+2184.690858321" watchObservedRunningTime="2026-10-31 21:52:24.56774979 +0000 UTC m=+2184.692857021" Oct 31 21:52:24.687989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.687962 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:24.740628 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.740598 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a398ee-a7cc-4921-8142-948bc6809846-kube-api-access-85856\" (UniqueName: \"kubernetes.io/projected/c2a398ee-a7cc-4921-8142-948bc6809846-kube-api-access-85856\") pod \"c2a398ee-a7cc-4921-8142-948bc6809846\" (UID: \"c2a398ee-a7cc-4921-8142-948bc6809846\") " Oct 31 21:52:24.742714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.742650 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a398ee-a7cc-4921-8142-948bc6809846-kube-api-access-85856" pod "c2a398ee-a7cc-4921-8142-948bc6809846" (UID: "c2a398ee-a7cc-4921-8142-948bc6809846"). InnerVolumeSpecName "kube-api-access-85856". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:52:24.842864 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:24.842836 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85856\" (UniqueName: \"kubernetes.io/projected/c2a398ee-a7cc-4921-8142-948bc6809846-kube-api-access-85856\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:25.532283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.532245 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7552-fd65ec"] Oct 31 21:52:25.540679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.540643 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7552-fd65ec"] Oct 31 21:52:25.546432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.546400 4294 generic.go:356] "Generic (PLEG): container finished" podID="c2a398ee-a7cc-4921-8142-948bc6809846" containerID="24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee" exitCode=137 Oct 31 21:52:25.546707 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.546687 4294 scope.go:122] "RemoveContainer" containerID="24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee" Oct 31 21:52:25.547137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.547115 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:25.557611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.557593 4294 scope.go:122] "RemoveContainer" containerID="24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee" Oct 31 21:52:25.557876 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:25.557852 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee\": container with ID starting with 24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee not found: ID does not exist" containerID="24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee" Oct 31 21:52:25.557977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.557888 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee"} err="failed to get container status \"24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee\": rpc error: code = NotFound desc = could not find container \"24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee\": container with ID starting with 24d8a7f4c4c4c0fb3a15e08538324245d7e416ac1caa072707c9a584684b2dee not found: ID does not exist" Oct 31 21:52:25.572990 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:25.572958 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7552-4ccea5\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="c2a398ee-a7cc-4921-8142-948bc6809846" pod="opendatahub/test-curl-7552-4ccea5" Oct 31 21:52:25.985769 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:25.985737 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a398ee-a7cc-4921-8142-948bc6809846" path="/var/lib/kubelet/pods/c2a398ee-a7cc-4921-8142-948bc6809846/volumes" Oct 31 21:52:26.557289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:26.557263 4294 generic.go:356] "Generic (PLEG): container finished" podID="b2352bf9-6de7-42af-a6b7-cab92c8c915e" containerID="88ec132e0f27f071600995ec44550565d447dc75fd3b044dca4ef1f5bdf97dee" exitCode=137 Oct 31 21:52:26.558617 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:26.558590 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7552-fd65ec" podUID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" containerName="test-curl-7552-fd65ec" containerID="cri-o://07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b" gracePeriod=2 Oct 31 21:52:26.704988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:26.704964 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:26.768665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:26.768634 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2352bf9-6de7-42af-a6b7-cab92c8c915e-kube-api-access-tm9qz\" (UniqueName: \"kubernetes.io/projected/b2352bf9-6de7-42af-a6b7-cab92c8c915e-kube-api-access-tm9qz\") pod \"b2352bf9-6de7-42af-a6b7-cab92c8c915e\" (UID: \"b2352bf9-6de7-42af-a6b7-cab92c8c915e\") " Oct 31 21:52:26.770727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:26.770690 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2352bf9-6de7-42af-a6b7-cab92c8c915e-kube-api-access-tm9qz" pod "b2352bf9-6de7-42af-a6b7-cab92c8c915e" (UID: "b2352bf9-6de7-42af-a6b7-cab92c8c915e"). InnerVolumeSpecName "kube-api-access-tm9qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:52:26.871155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:26.871083 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm9qz\" (UniqueName: \"kubernetes.io/projected/b2352bf9-6de7-42af-a6b7-cab92c8c915e-kube-api-access-tm9qz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:27.033716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:27.033675 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8"] Oct 31 21:52:27.046604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:27.046572 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b54a35fc-s4dv8"] Oct 31 21:52:27.568351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:27.568325 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:27.568766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:27.568298 4294 scope.go:122] "RemoveContainer" containerID="88ec132e0f27f071600995ec44550565d447dc75fd3b044dca4ef1f5bdf97dee" Oct 31 21:52:27.589619 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:27.589590 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7552-c259b0\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="b2352bf9-6de7-42af-a6b7-cab92c8c915e" pod="opendatahub/test-curl-7552-c259b0" Oct 31 21:52:27.987139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:27.987096 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936fa442-323f-4f8a-9c10-523bd7ec6475" path="/var/lib/kubelet/pods/936fa442-323f-4f8a-9c10-523bd7ec6475/volumes" Oct 31 21:52:27.987575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:27.987557 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2352bf9-6de7-42af-a6b7-cab92c8c915e" path="/var/lib/kubelet/pods/b2352bf9-6de7-42af-a6b7-cab92c8c915e/volumes" Oct 31 21:52:28.786048 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:28.785989 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:28.910711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:28.905293 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee38fa5a-d473-4a99-8a14-4050b0224e3e-kube-api-access-cgr28\" (UniqueName: \"kubernetes.io/projected/ee38fa5a-d473-4a99-8a14-4050b0224e3e-kube-api-access-cgr28\") pod \"ee38fa5a-d473-4a99-8a14-4050b0224e3e\" (UID: \"ee38fa5a-d473-4a99-8a14-4050b0224e3e\") " Oct 31 21:52:28.921382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:28.915488 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee38fa5a-d473-4a99-8a14-4050b0224e3e-kube-api-access-cgr28" pod "ee38fa5a-d473-4a99-8a14-4050b0224e3e" (UID: "ee38fa5a-d473-4a99-8a14-4050b0224e3e"). InnerVolumeSpecName "kube-api-access-cgr28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:52:29.007791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:29.007758 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr28\" (UniqueName: \"kubernetes.io/projected/ee38fa5a-d473-4a99-8a14-4050b0224e3e-kube-api-access-cgr28\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:52:29.586004 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:29.585970 4294 generic.go:356] "Generic (PLEG): container finished" podID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" containerID="07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b" exitCode=137 Oct 31 21:52:29.586183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:29.586019 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:29.586183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:29.586031 4294 scope.go:122] "RemoveContainer" containerID="07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b" Oct 31 21:52:29.595955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:29.595931 4294 scope.go:122] "RemoveContainer" containerID="07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b" Oct 31 21:52:29.596225 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:29.596199 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b\": container with ID starting with 07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b not found: ID does not exist" containerID="07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b" Oct 31 21:52:29.596325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:29.596225 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b"} err="failed to get container status \"07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b\": rpc error: code = NotFound desc = could not find container \"07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b\": container with ID starting with 07ba9fb8b086f0015b40cb02a1b69610e9b43de5d7caed532935568aa15c004b not found: ID does not exist" Oct 31 21:52:29.607505 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:29.607479 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7552-fd65ec\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:29.985240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:29.985202 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" path="/var/lib/kubelet/pods/ee38fa5a-d473-4a99-8a14-4050b0224e3e/volumes" Oct 31 21:52:29.994917 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:29.994879 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7552-fd65ec\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" pod="opendatahub/test-curl-7552-fd65ec" Oct 31 21:52:31.310867 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:52:31.310832 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:52:34.033222 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:34.033188 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl"] Oct 31 21:52:34.045142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:34.045116 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5790797-sknwl"] Oct 31 21:52:35.985148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:35.985111 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36701014-e0e5-4f90-a53f-4de09038bbe3" path="/var/lib/kubelet/pods/36701014-e0e5-4f90-a53f-4de09038bbe3/volumes" Oct 31 21:52:41.035188 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:41.035153 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q"] Oct 31 21:52:41.046783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:41.046753 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh"] Oct 31 21:52:41.049526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:41.049510 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73d3dd93-bxg5q"] Oct 31 21:52:41.052089 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:41.052067 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8ba20b-tlkgh"] Oct 31 21:52:41.984909 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:41.984875 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ce57a0-d866-41c4-840a-cad25c4c46e0" path="/var/lib/kubelet/pods/62ce57a0-d866-41c4-840a-cad25c4c46e0/volumes" Oct 31 21:52:41.985351 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:52:41.985338 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb753653-a7e5-46cc-aaf8-d2ce11439a00" path="/var/lib/kubelet/pods/fb753653-a7e5-46cc-aaf8-d2ce11439a00/volumes" Oct 31 21:53:01.315606 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:01.315520 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:53:01.484256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.484222 4294 scope.go:122] "RemoveContainer" containerID="5497b9698038b76bf3faff4a8b6cb4a299483cce5b2516de166b67bc13861a85" Oct 31 21:53:01.494086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.494053 4294 scope.go:122] "RemoveContainer" containerID="a37b26562fd7c5a2691f1c27275e56645ed0c52b8227dc6128c075e55ee06c00" Oct 31 21:53:01.503126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.503103 4294 scope.go:122] "RemoveContainer" containerID="e7941b04b3a03919f3f880b74c77fabc87234e98af57e0d3a7da8a97ee94ea86" Oct 31 21:53:01.514598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.514578 4294 scope.go:122] "RemoveContainer" containerID="f0def5150250ad8b6e064a8e17dce6eada1afa9b8b7599dbe5ff87213a22a97a" Oct 31 21:53:01.523497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.523472 4294 scope.go:122] "RemoveContainer" containerID="6124eeeefb2c8e69703a3ff58d5eeaa3752a306dcd0c533d640b23edd4635e47" Oct 31 21:53:01.531802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.531780 4294 scope.go:122] "RemoveContainer" containerID="b4a532e8565aa03496d6dbd8873c3db4fc9421ddb50d36b650e26ffe16752a14" Oct 31 21:53:01.540122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.540102 4294 scope.go:122] "RemoveContainer" containerID="cc27606128da81628d0f6484775ba6ca3befb62853f8a4d858464f28a30891e5" Oct 31 21:53:01.548212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:01.548190 4294 scope.go:122] "RemoveContainer" containerID="5f6db778fef008a38791068293370ee9454499beab7175762e80d435ee090aae" Oct 31 21:53:23.079072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.079040 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2"] Oct 31 21:53:23.079539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.079258 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" containerID="cri-o://59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6" gracePeriod=30 Oct 31 21:53:23.091917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.091891 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k"] Oct 31 21:53:23.092133 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.092109 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" containerID="cri-o://9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39" gracePeriod=30 Oct 31 21:53:23.095357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.095329 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl"] Oct 31 21:53:23.095572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.095545 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" containerID="cri-o://f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b" gracePeriod=30 Oct 31 21:53:23.104055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.104026 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl"] Oct 31 21:53:23.104369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:23.104280 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" containerID="cri-o://d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac" gracePeriod=30 Oct 31 21:53:28.410507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.410472 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:53:28.481901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.481866 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:53:28.487033 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.487000 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:53:28.495254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.494464 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576451 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-credential-socket\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576506 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-envoy\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576548 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-data\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576574 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-socket\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576606 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-kube-api-access-ngf5r\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-kube-api-access-ngf5r\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576644 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-envoy\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576720 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-data\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576750 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-envoy\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576787 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583ea135-1351-4fd2-a308-42bf4fab268e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583ea135-1351-4fd2-a308-42bf4fab268e-istiod-ca-cert\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576837 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-kube-api-access-zbnvn\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-kube-api-access-zbnvn\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576872 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-certs\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576903 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408faa8a-75a3-4f10-babf-6de042231a88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408faa8a-75a3-4f10-babf-6de042231a88-istio-podinfo\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.576937 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istiod-ca-cert\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.584892 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-envoy" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.585131 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-credential-socket" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.585472 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583ea135-1351-4fd2-a308-42bf4fab268e-istiod-ca-cert" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:28.586437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.585699 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-data" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.586601 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-certs" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.586690 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-credential-socket\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.586757 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-token\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-token\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.586791 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-socket\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.586866 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-podinfo\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.586918 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-kube-api-access-vdhzh\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-kube-api-access-vdhzh\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.586981 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-certs\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587013 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-socket" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587021 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-credential-socket\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587070 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-data\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587105 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-credential-socket\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587147 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-socket\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587174 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-istio-token\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-istio-token\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587215 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-socket" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587224 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-istio-token\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-istio-token\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587276 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/583ea135-1351-4fd2-a308-42bf4fab268e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583ea135-1351-4fd2-a308-42bf4fab268e-istio-podinfo\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587349 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43407457-00de-4bac-87a6-9842f359dbb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43407457-00de-4bac-87a6-9842f359dbb5-istiod-ca-cert\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587390 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43407457-00de-4bac-87a6-9842f359dbb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43407457-00de-4bac-87a6-9842f359dbb5-istio-podinfo\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587445 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-kube-api-access-jvjbd\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-kube-api-access-jvjbd\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587476 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-certs\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587505 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-data\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.587571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587553 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-certs\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587601 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-istio-token\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-istio-token\") pod \"583ea135-1351-4fd2-a308-42bf4fab268e\" (UID: \"583ea135-1351-4fd2-a308-42bf4fab268e\") " Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587632 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408faa8a-75a3-4f10-babf-6de042231a88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408faa8a-75a3-4f10-babf-6de042231a88-istiod-ca-cert\") pod \"408faa8a-75a3-4f10-babf-6de042231a88\" (UID: \"408faa8a-75a3-4f10-babf-6de042231a88\") " Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587663 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-envoy\") pod \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\" (UID: \"de4662f0-9136-44ad-8d6d-fe0ea091fafe\") " Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587694 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-socket\") pod \"43407457-00de-4bac-87a6-9842f359dbb5\" (UID: \"43407457-00de-4bac-87a6-9842f359dbb5\") " Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587470 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-envoy" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588000 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-data" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588180 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-socket" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588192 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-data" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588668 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588691 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588708 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.588708 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588722 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588738 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588751 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588765 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588778 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588791 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588805 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/583ea135-1351-4fd2-a308-42bf4fab268e-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.588818 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.589037 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-data" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.589369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.589214 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-certs" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.591787 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.591760 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-istio-token" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.592932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.592073 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408faa8a-75a3-4f10-babf-6de042231a88-istiod-ca-cert" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:28.592932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.592289 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-token" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.593377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.593164 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-credential-socket" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.593747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.593550 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-certs" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.593747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587514 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istiod-ca-cert" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:28.593747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587667 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-credential-socket" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.593747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.587972 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-envoy" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.593976 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.593904 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-socket" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.595370 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.595313 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-kube-api-access-jvjbd" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "kube-api-access-jvjbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.595467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.595386 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/583ea135-1351-4fd2-a308-42bf4fab268e-istio-podinfo" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:53:28.595467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.595425 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43407457-00de-4bac-87a6-9842f359dbb5-istio-podinfo" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:53:28.595590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.595562 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-certs" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.596044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.596016 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-credential-socket" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.596136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.596119 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43407457-00de-4bac-87a6-9842f359dbb5-istiod-ca-cert" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:28.596278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.596258 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-istio-token" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.596278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.596262 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-envoy" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:53:28.597513 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.597492 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-istio-token" pod "43407457-00de-4bac-87a6-9842f359dbb5" (UID: "43407457-00de-4bac-87a6-9842f359dbb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.597855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.597824 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-kube-api-access-vdhzh" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "kube-api-access-vdhzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.603050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.603021 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408faa8a-75a3-4f10-babf-6de042231a88-istio-podinfo" pod "408faa8a-75a3-4f10-babf-6de042231a88" (UID: "408faa8a-75a3-4f10-babf-6de042231a88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:53:28.607606 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.607570 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-kube-api-access-zbnvn" pod "583ea135-1351-4fd2-a308-42bf4fab268e" (UID: "583ea135-1351-4fd2-a308-42bf4fab268e"). InnerVolumeSpecName "kube-api-access-zbnvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.608063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.608015 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb"] Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609227 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" containerName="test-curl-7552-fd65ec" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609250 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" containerName="test-curl-7552-fd65ec" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609278 4294 cpu_manager.go:414] "containerMap: removing container" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609285 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609324 4294 cpu_manager.go:414] "containerMap: removing container" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609331 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609346 4294 cpu_manager.go:414] "containerMap: removing container" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609353 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609409 4294 cpu_manager.go:414] "containerMap: removing container" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609416 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609550 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee38fa5a-d473-4a99-8a14-4050b0224e3e" containerName="test-curl-7552-fd65ec" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609564 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609577 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="43407457-00de-4bac-87a6-9842f359dbb5" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609587 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609602 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" Oct 31 21:53:28.612356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.609611 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="408faa8a-75a3-4f10-babf-6de042231a88" containerName="istio-proxy" Oct 31 21:53:28.619050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.618372 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-podinfo" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:53:28.647743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.627427 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-kube-api-access-ngf5r" pod "de4662f0-9136-44ad-8d6d-fe0ea091fafe" (UID: "de4662f0-9136-44ad-8d6d-fe0ea091fafe"). InnerVolumeSpecName "kube-api-access-ngf5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:28.647743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.638602 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb"] Oct 31 21:53:28.647743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.638714 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.659922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.655911 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-c34cbd-openshift-default-dockercfg-wmpbh\"" Oct 31 21:53:28.690523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690489 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/583ea135-1351-4fd2-a308-42bf4fab268e-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690523 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43407457-00de-4bac-87a6-9842f359dbb5-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690534 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43407457-00de-4bac-87a6-9842f359dbb5-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690543 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvjbd\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-kube-api-access-jvjbd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690552 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690562 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690575 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690587 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690599 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408faa8a-75a3-4f10-babf-6de042231a88-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690610 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690623 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngf5r\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-kube-api-access-ngf5r\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690637 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/583ea135-1351-4fd2-a308-42bf4fab268e-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690650 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbnvn\" (UniqueName: \"kubernetes.io/projected/583ea135-1351-4fd2-a308-42bf4fab268e-kube-api-access-zbnvn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690663 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408faa8a-75a3-4f10-babf-6de042231a88-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690675 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690688 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690700 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690712 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de4662f0-9136-44ad-8d6d-fe0ea091fafe-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690724 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhzh\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-kube-api-access-vdhzh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690737 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690750 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408faa8a-75a3-4f10-babf-6de042231a88-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690764 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43407457-00de-4bac-87a6-9842f359dbb5-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.690763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690773 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de4662f0-9136-44ad-8d6d-fe0ea091fafe-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.691887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690781 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408faa8a-75a3-4f10-babf-6de042231a88-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.691887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.690789 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43407457-00de-4bac-87a6-9842f359dbb5-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795052 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-certs\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795124 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-envoy\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795239 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istiod-ca-cert\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795281 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-data\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795342 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-podinfo\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795417 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8pp4\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-kube-api-access-j8pp4\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795452 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-credential-socket\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795488 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-socket\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.799325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.795535 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-token\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897128 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-certs\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897181 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-envoy\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897247 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istiod-ca-cert\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897272 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-data\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897316 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-podinfo\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897370 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8pp4\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-kube-api-access-j8pp4\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897391 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-credential-socket\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897411 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-socket\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.897870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.897441 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-token\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.908362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.898119 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-certs\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.908362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.899018 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-credential-socket\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.908362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.899598 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-socket\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.908362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.899700 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-data\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.908362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.900342 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istiod-ca-cert\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.914607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.914576 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-envoy\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.916176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.916148 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-podinfo\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.936082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.935636 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-token\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.936696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.936662 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8pp4\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-kube-api-access-j8pp4\") pod \"e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:28.966239 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:28.966206 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:29.061485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.061413 4294 generic.go:356] "Generic (PLEG): container finished" podID="583ea135-1351-4fd2-a308-42bf4fab268e" containerID="9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39" exitCode=0 Oct 31 21:53:29.061665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.061644 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" Oct 31 21:53:29.062613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.062581 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" event={"ID":"583ea135-1351-4fd2-a308-42bf4fab268e","Type":"ContainerDied","Data":"9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39"} Oct 31 21:53:29.062737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.062646 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k" event={"ID":"583ea135-1351-4fd2-a308-42bf4fab268e","Type":"ContainerDied","Data":"8cb53353e8c25404ee0ec15e703337eed8054e4773eed4ecae1b58a2a242e6c9"} Oct 31 21:53:29.062737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.062676 4294 scope.go:122] "RemoveContainer" containerID="9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39" Oct 31 21:53:29.076336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.071746 4294 generic.go:356] "Generic (PLEG): container finished" podID="43407457-00de-4bac-87a6-9842f359dbb5" containerID="d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac" exitCode=0 Oct 31 21:53:29.076336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.071862 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" event={"ID":"43407457-00de-4bac-87a6-9842f359dbb5","Type":"ContainerDied","Data":"d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac"} Oct 31 21:53:29.076336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.071907 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" event={"ID":"43407457-00de-4bac-87a6-9842f359dbb5","Type":"ContainerDied","Data":"10fcbbce325acd3731c96761fe011c29116a623f67c89c03effcd8817579979c"} Oct 31 21:53:29.076336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.071988 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl" Oct 31 21:53:29.076630 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.076528 4294 generic.go:356] "Generic (PLEG): container finished" podID="408faa8a-75a3-4f10-babf-6de042231a88" containerID="59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6" exitCode=0 Oct 31 21:53:29.077395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.076713 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" event={"ID":"408faa8a-75a3-4f10-babf-6de042231a88","Type":"ContainerDied","Data":"59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6"} Oct 31 21:53:29.077395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.076760 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" event={"ID":"408faa8a-75a3-4f10-babf-6de042231a88","Type":"ContainerDied","Data":"71b5a2a9dc706196caa28a7fa050eb8913145fb5c54f0d339580fd792c0c4b8e"} Oct 31 21:53:29.077395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.076881 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2" Oct 31 21:53:29.082509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.082484 4294 scope.go:122] "RemoveContainer" containerID="9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39" Oct 31 21:53:29.093615 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:29.088099 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39\": container with ID starting with 9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39 not found: ID does not exist" containerID="9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39" Oct 31 21:53:29.093615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.088156 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39"} err="failed to get container status \"9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39\": rpc error: code = NotFound desc = could not find container \"9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39\": container with ID starting with 9719f514cf9d14e91c2c710011ee473680901ac8e2211e68e3af91fab6ff2d39 not found: ID does not exist" Oct 31 21:53:29.093615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.088281 4294 scope.go:122] "RemoveContainer" containerID="d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac" Oct 31 21:53:29.094467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.094386 4294 generic.go:356] "Generic (PLEG): container finished" podID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerID="f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b" exitCode=0 Oct 31 21:53:29.094525 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.094468 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" event={"ID":"de4662f0-9136-44ad-8d6d-fe0ea091fafe","Type":"ContainerDied","Data":"f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b"} Oct 31 21:53:29.094584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.094560 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" event={"ID":"de4662f0-9136-44ad-8d6d-fe0ea091fafe","Type":"ContainerDied","Data":"7f418c14df59f0e5c2f1dba9200acf368da1c4a67f2c81e8821b0c9ae7fb235f"} Oct 31 21:53:29.094731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.094710 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl" Oct 31 21:53:29.159808 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.159785 4294 scope.go:122] "RemoveContainer" containerID="d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac" Oct 31 21:53:29.160392 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:29.160359 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac\": container with ID starting with d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac not found: ID does not exist" containerID="d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac" Oct 31 21:53:29.160555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.160526 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac"} err="failed to get container status \"d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac\": rpc error: code = NotFound desc = could not find container \"d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac\": container with ID starting with d918b2470b6873df62f60f497a07b01414fe1b5cbaf49574982a10dd9c2d43ac not found: ID does not exist" Oct 31 21:53:29.160555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.160554 4294 scope.go:122] "RemoveContainer" containerID="59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6" Oct 31 21:53:29.164781 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.164759 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k"] Oct 31 21:53:29.168754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.168728 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xvl7k"] Oct 31 21:53:29.173326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.173281 4294 scope.go:122] "RemoveContainer" containerID="59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6" Oct 31 21:53:29.173609 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:29.173588 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6\": container with ID starting with 59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6 not found: ID does not exist" containerID="59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6" Oct 31 21:53:29.173661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.173618 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6"} err="failed to get container status \"59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6\": rpc error: code = NotFound desc = could not find container \"59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6\": container with ID starting with 59cbbeb4ab671856b9cd8d896f0b6648001ec45a64074be63419d94e9ce5b7c6 not found: ID does not exist" Oct 31 21:53:29.173661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.173637 4294 scope.go:122] "RemoveContainer" containerID="f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b" Oct 31 21:53:29.183030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.183011 4294 scope.go:122] "RemoveContainer" containerID="65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e" Oct 31 21:53:29.192100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.192082 4294 scope.go:122] "RemoveContainer" containerID="f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b" Oct 31 21:53:29.192416 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:29.192395 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b\": container with ID starting with f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b not found: ID does not exist" containerID="f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b" Oct 31 21:53:29.192511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.192423 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b"} err="failed to get container status \"f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b\": rpc error: code = NotFound desc = could not find container \"f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b\": container with ID starting with f045bbb705f611eec707ce529697a4c7207716d0692714e08d351b035575132b not found: ID does not exist" Oct 31 21:53:29.192511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.192441 4294 scope.go:122] "RemoveContainer" containerID="65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e" Oct 31 21:53:29.192714 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:29.192694 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e\": container with ID starting with 65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e not found: ID does not exist" containerID="65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e" Oct 31 21:53:29.192772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.192719 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e"} err="failed to get container status \"65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e\": rpc error: code = NotFound desc = could not find container \"65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e\": container with ID starting with 65d7716c58799db11a59635cf59c2f6c965cdfa11a1ed501c0516c8b53dd240e not found: ID does not exist" Oct 31 21:53:29.215903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.215873 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl"] Oct 31 21:53:29.219478 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.219457 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-zhsnl"] Oct 31 21:53:29.253903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.253880 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl"] Oct 31 21:53:29.261888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.261864 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-bkzzl"] Oct 31 21:53:29.295937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.295911 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2"] Oct 31 21:53:29.299918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.299893 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb"] Oct 31 21:53:29.300974 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:53:29.300947 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba9dfb2_596f_4ade_85eb_ca423cbbd1ab.slice/crio-a64559a8c370ee08b88c2f28549d290676f86b58220dad2cce35c9e75765e333 WatchSource:0}: Error finding container a64559a8c370ee08b88c2f28549d290676f86b58220dad2cce35c9e75765e333: Status 404 returned error can't find the container with id a64559a8c370ee08b88c2f28549d290676f86b58220dad2cce35c9e75765e333 Oct 31 21:53:29.302580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.302559 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-cc45a7-openshift-default-55cdb6f7d7-xspc2"] Oct 31 21:53:29.303852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.303811 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:53:29.304067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.303881 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:53:29.304067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.303909 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:53:29.985379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.985345 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408faa8a-75a3-4f10-babf-6de042231a88" path="/var/lib/kubelet/pods/408faa8a-75a3-4f10-babf-6de042231a88/volumes" Oct 31 21:53:29.985863 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.985848 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43407457-00de-4bac-87a6-9842f359dbb5" path="/var/lib/kubelet/pods/43407457-00de-4bac-87a6-9842f359dbb5/volumes" Oct 31 21:53:29.986368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.986333 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583ea135-1351-4fd2-a308-42bf4fab268e" path="/var/lib/kubelet/pods/583ea135-1351-4fd2-a308-42bf4fab268e/volumes" Oct 31 21:53:29.987103 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:29.987086 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" path="/var/lib/kubelet/pods/de4662f0-9136-44ad-8d6d-fe0ea091fafe/volumes" Oct 31 21:53:30.105609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:30.105575 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" event={"ID":"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab","Type":"ContainerStarted","Data":"4309e06a403166918045e0265f431e08c7791ecf4c98f85bad798b8d1846c8ba"} Oct 31 21:53:30.105816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:30.105632 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" event={"ID":"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab","Type":"ContainerStarted","Data":"a64559a8c370ee08b88c2f28549d290676f86b58220dad2cce35c9e75765e333"} Oct 31 21:53:30.166293 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:30.166239 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podStartSLOduration=2.16622349 podStartE2EDuration="2.16622349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:53:30.163907 +0000 UTC m=+2250.289014231" watchObservedRunningTime="2026-10-31 21:53:30.16622349 +0000 UTC m=+2250.291330720" Oct 31 21:53:30.966375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:30.966341 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:30.972240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:30.972213 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:31.113811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:31.113778 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:31.114753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:31.114737 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:53:31.313509 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:31.313473 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:53:39.865189 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.865156 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr"] Oct 31 21:53:39.865734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.865720 4294 cpu_manager.go:414] "containerMap: removing container" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" Oct 31 21:53:39.865782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.865735 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4662f0-9136-44ad-8d6d-fe0ea091fafe" containerName="istio-proxy" Oct 31 21:53:39.870558 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.870541 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:39.892402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.892373 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr"] Oct 31 21:53:39.900143 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.900119 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:53:39.900358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.900340 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:53:39.971253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.971223 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:39.971450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.971378 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:39.971555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:39.971539 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk6ff\" (UniqueName: \"kubernetes.io/projected/400a2227-c7be-4132-b9db-609d9434df1f-kube-api-access-kk6ff\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.073474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.073430 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk6ff\" (UniqueName: \"kubernetes.io/projected/400a2227-c7be-4132-b9db-609d9434df1f-kube-api-access-kk6ff\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.073755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.073568 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.073755 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.073708 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.074538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.074513 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.078475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.078447 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.098016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.097981 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk6ff\" (UniqueName: \"kubernetes.io/projected/400a2227-c7be-4132-b9db-609d9434df1f-kube-api-access-kk6ff\") pod \"maas-api-revoke-sub-ec1ab11b-q2ccr\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.183326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.183226 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:40.193264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.193243 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7546-35d504"] Oct 31 21:53:40.197160 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.197144 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:40.215262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.212396 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dsnkq\"" Oct 31 21:53:40.215262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.214097 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-35d504"] Oct 31 21:53:40.282828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.278550 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwg6\" (UniqueName: \"kubernetes.io/projected/fff879a9-687a-48dc-b16b-edd4de1c1ae9-kube-api-access-qvwg6\") pod \"test-curl-7546-35d504\" (UID: \"fff879a9-687a-48dc-b16b-edd4de1c1ae9\") " pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:40.380278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.380250 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwg6\" (UniqueName: \"kubernetes.io/projected/fff879a9-687a-48dc-b16b-edd4de1c1ae9-kube-api-access-qvwg6\") pod \"test-curl-7546-35d504\" (UID: \"fff879a9-687a-48dc-b16b-edd4de1c1ae9\") " pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:40.403524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.403498 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwg6\" (UniqueName: \"kubernetes.io/projected/fff879a9-687a-48dc-b16b-edd4de1c1ae9-kube-api-access-qvwg6\") pod \"test-curl-7546-35d504\" (UID: \"fff879a9-687a-48dc-b16b-edd4de1c1ae9\") " pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:40.499928 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.499898 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr"] Oct 31 21:53:40.500609 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:53:40.500580 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400a2227_c7be_4132_b9db_609d9434df1f.slice/crio-8cdb56b4626d2298d138a64211a67f579ae94d6c5a8a87ec1d1452a010623913 WatchSource:0}: Error finding container 8cdb56b4626d2298d138a64211a67f579ae94d6c5a8a87ec1d1452a010623913: Status 404 returned error can't find the container with id 8cdb56b4626d2298d138a64211a67f579ae94d6c5a8a87ec1d1452a010623913 Oct 31 21:53:40.542212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.542187 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:40.847429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:40.847396 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7546-35d504"] Oct 31 21:53:40.847677 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:53:40.847655 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfff879a9_687a_48dc_b16b_edd4de1c1ae9.slice/crio-85cbaa8c5cfee88f60a753c67bfccf4dc2bfaa1d21a6a2078361dbb835f17a0f WatchSource:0}: Error finding container 85cbaa8c5cfee88f60a753c67bfccf4dc2bfaa1d21a6a2078361dbb835f17a0f: Status 404 returned error can't find the container with id 85cbaa8c5cfee88f60a753c67bfccf4dc2bfaa1d21a6a2078361dbb835f17a0f Oct 31 21:53:41.249855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:41.249817 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-35d504" event={"ID":"fff879a9-687a-48dc-b16b-edd4de1c1ae9","Type":"ContainerStarted","Data":"2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254"} Oct 31 21:53:41.250358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:41.249867 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7546-35d504" event={"ID":"fff879a9-687a-48dc-b16b-edd4de1c1ae9","Type":"ContainerStarted","Data":"85cbaa8c5cfee88f60a753c67bfccf4dc2bfaa1d21a6a2078361dbb835f17a0f"} Oct 31 21:53:41.251490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:41.251461 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" event={"ID":"400a2227-c7be-4132-b9db-609d9434df1f","Type":"ContainerStarted","Data":"91b1e77300a531f7d3c9e59b42f085638396282a0540841c08a73850e66a4626"} Oct 31 21:53:41.251603 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:41.251512 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" event={"ID":"400a2227-c7be-4132-b9db-609d9434df1f","Type":"ContainerStarted","Data":"8cdb56b4626d2298d138a64211a67f579ae94d6c5a8a87ec1d1452a010623913"} Oct 31 21:53:41.288727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:41.288681 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7546-35d504" podStartSLOduration=1.288668594 podStartE2EDuration="1.288668594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:53:41.287279498 +0000 UTC m=+2261.412386711" watchObservedRunningTime="2026-10-31 21:53:41.288668594 +0000 UTC m=+2261.413775824" Oct 31 21:53:41.332711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:41.332670 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" podStartSLOduration=2.332656761 podStartE2EDuration="2.332656761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:53:41.329849527 +0000 UTC m=+2261.454956763" watchObservedRunningTime="2026-10-31 21:53:41.332656761 +0000 UTC m=+2261.457763991" Oct 31 21:53:42.138613 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.138580 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7546-35d504"] Oct 31 21:53:42.143157 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.143121 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7546-35d504"] Oct 31 21:53:42.259345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.259314 4294 generic.go:356] "Generic (PLEG): container finished" podID="400a2227-c7be-4132-b9db-609d9434df1f" containerID="91b1e77300a531f7d3c9e59b42f085638396282a0540841c08a73850e66a4626" exitCode=0 Oct 31 21:53:42.259788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.259395 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" event={"ID":"400a2227-c7be-4132-b9db-609d9434df1f","Type":"ContainerDied","Data":"91b1e77300a531f7d3c9e59b42f085638396282a0540841c08a73850e66a4626"} Oct 31 21:53:42.350283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.350252 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h"] Oct 31 21:53:42.351151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.351126 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" containerName="test-curl-7546-35d504" Oct 31 21:53:42.351151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.351149 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" containerName="test-curl-7546-35d504" Oct 31 21:53:42.351381 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.351335 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" containerName="test-curl-7546-35d504" Oct 31 21:53:42.355393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.355366 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.372660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.372635 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h"] Oct 31 21:53:42.405280 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.405199 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.405459 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.405403 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnmg\" (UniqueName: \"kubernetes.io/projected/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-kube-api-access-hxnmg\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.405687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.405665 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.507446 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.507403 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.507654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.507566 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.507729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.507672 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnmg\" (UniqueName: \"kubernetes.io/projected/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-kube-api-access-hxnmg\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.508114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.508092 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.509923 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.509902 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.530890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.530855 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnmg\" (UniqueName: \"kubernetes.io/projected/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-kube-api-access-hxnmg\") pod \"maas-api-revoke-sub-d85631e7-n2v2h\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.669780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.669702 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:42.973571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:42.973540 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h"] Oct 31 21:53:42.980255 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:53:42.980228 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7781c3_0e96_4dbc_ad4a_f3c9fb69d4b4.slice/crio-49f2b784e4607db61b3cacdfa9601423d42fffe825545ded83ab58dadedcc65d WatchSource:0}: Error finding container 49f2b784e4607db61b3cacdfa9601423d42fffe825545ded83ab58dadedcc65d: Status 404 returned error can't find the container with id 49f2b784e4607db61b3cacdfa9601423d42fffe825545ded83ab58dadedcc65d Oct 31 21:53:43.271346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.271269 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7546-35d504" podUID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" containerName="test-curl-7546-35d504" containerID="cri-o://2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254" gracePeriod=2 Oct 31 21:53:43.271874 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.271574 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" event={"ID":"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4","Type":"ContainerStarted","Data":"49f2b784e4607db61b3cacdfa9601423d42fffe825545ded83ab58dadedcc65d"} Oct 31 21:53:43.287501 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:43.287452 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7546-35d504\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:43.424390 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.424362 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:43.478450 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:43.478413 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7546-35d504\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:43.528126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.528072 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-openshift-ca\") pod \"400a2227-c7be-4132-b9db-609d9434df1f\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " Oct 31 21:53:43.528234 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.528218 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400a2227-c7be-4132-b9db-609d9434df1f-kube-api-access-kk6ff\" (UniqueName: \"kubernetes.io/projected/400a2227-c7be-4132-b9db-609d9434df1f-kube-api-access-kk6ff\") pod \"400a2227-c7be-4132-b9db-609d9434df1f\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " Oct 31 21:53:43.528275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.528240 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-xks-ca\") pod \"400a2227-c7be-4132-b9db-609d9434df1f\" (UID: \"400a2227-c7be-4132-b9db-609d9434df1f\") " Oct 31 21:53:43.528444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.528421 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-openshift-ca" pod "400a2227-c7be-4132-b9db-609d9434df1f" (UID: "400a2227-c7be-4132-b9db-609d9434df1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:43.528766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.528738 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:43.530233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.530212 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-xks-ca" pod "400a2227-c7be-4132-b9db-609d9434df1f" (UID: "400a2227-c7be-4132-b9db-609d9434df1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:53:43.530371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.530349 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400a2227-c7be-4132-b9db-609d9434df1f-kube-api-access-kk6ff" pod "400a2227-c7be-4132-b9db-609d9434df1f" (UID: "400a2227-c7be-4132-b9db-609d9434df1f"). InnerVolumeSpecName "kube-api-access-kk6ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:43.630580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.630537 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk6ff\" (UniqueName: \"kubernetes.io/projected/400a2227-c7be-4132-b9db-609d9434df1f-kube-api-access-kk6ff\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:43.630580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:43.630577 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/400a2227-c7be-4132-b9db-609d9434df1f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:44.280087 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:44.280053 4294 generic.go:356] "Generic (PLEG): container finished" podID="5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" containerID="d8d96d68da340cec2e0d6a455f73614ab093bce47ab733ec7774ee3f3211dda6" exitCode=0 Oct 31 21:53:44.280526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:44.280139 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" event={"ID":"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4","Type":"ContainerDied","Data":"d8d96d68da340cec2e0d6a455f73614ab093bce47ab733ec7774ee3f3211dda6"} Oct 31 21:53:44.281432 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:44.281412 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" Oct 31 21:53:44.281568 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:44.281438 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr" event={"ID":"400a2227-c7be-4132-b9db-609d9434df1f","Type":"ContainerDied","Data":"8cdb56b4626d2298d138a64211a67f579ae94d6c5a8a87ec1d1452a010623913"} Oct 31 21:53:44.281568 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:44.281476 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cdb56b4626d2298d138a64211a67f579ae94d6c5a8a87ec1d1452a010623913" Oct 31 21:53:45.469488 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.469465 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:45.472857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.472835 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:45.558085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.558009 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff879a9-687a-48dc-b16b-edd4de1c1ae9-kube-api-access-qvwg6\" (UniqueName: \"kubernetes.io/projected/fff879a9-687a-48dc-b16b-edd4de1c1ae9-kube-api-access-qvwg6\") pod \"fff879a9-687a-48dc-b16b-edd4de1c1ae9\" (UID: \"fff879a9-687a-48dc-b16b-edd4de1c1ae9\") " Oct 31 21:53:45.558085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.558049 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-xks-ca\") pod \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " Oct 31 21:53:45.558286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.558137 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-kube-api-access-hxnmg\" (UniqueName: \"kubernetes.io/projected/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-kube-api-access-hxnmg\") pod \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " Oct 31 21:53:45.558286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.558212 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-openshift-ca\") pod \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\" (UID: \"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4\") " Oct 31 21:53:45.558673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.558649 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-openshift-ca" pod "5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" (UID: "5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:45.558918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.558898 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:45.560398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.560372 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-kube-api-access-hxnmg" pod "5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" (UID: "5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4"). InnerVolumeSpecName "kube-api-access-hxnmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:45.560398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.560385 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-xks-ca" pod "5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" (UID: "5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:53:45.560525 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.560408 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff879a9-687a-48dc-b16b-edd4de1c1ae9-kube-api-access-qvwg6" pod "fff879a9-687a-48dc-b16b-edd4de1c1ae9" (UID: "fff879a9-687a-48dc-b16b-edd4de1c1ae9"). InnerVolumeSpecName "kube-api-access-qvwg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:45.660961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.660912 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvwg6\" (UniqueName: \"kubernetes.io/projected/fff879a9-687a-48dc-b16b-edd4de1c1ae9-kube-api-access-qvwg6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:45.660961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.660950 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:45.661190 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.660966 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnmg\" (UniqueName: \"kubernetes.io/projected/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4-kube-api-access-hxnmg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:45.986171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:45.986139 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" path="/var/lib/kubelet/pods/fff879a9-687a-48dc-b16b-edd4de1c1ae9/volumes" Oct 31 21:53:46.130113 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.130079 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz"] Oct 31 21:53:46.130674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.130656 4294 cpu_manager.go:414] "containerMap: removing container" podUID="400a2227-c7be-4132-b9db-609d9434df1f" containerName="revoke-keys" Oct 31 21:53:46.130674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.130673 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400a2227-c7be-4132-b9db-609d9434df1f" containerName="revoke-keys" Oct 31 21:53:46.130844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.130697 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" containerName="revoke-keys" Oct 31 21:53:46.130844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.130703 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" containerName="revoke-keys" Oct 31 21:53:46.130844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.130802 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="400a2227-c7be-4132-b9db-609d9434df1f" containerName="revoke-keys" Oct 31 21:53:46.130844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.130813 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" containerName="revoke-keys" Oct 31 21:53:46.134840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.134820 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.159174 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.150889 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-56a6d4-openshift-default-dockercfg-z6rgj\"" Oct 31 21:53:46.179352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.176288 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz"] Oct 31 21:53:46.277384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.276920 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7w6\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-kube-api-access-kd7w6\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.277384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.277043 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-podinfo\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.277384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.277081 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-data\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.277384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.277111 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-credential-socket\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.277384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.277188 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-token\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.277384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.277221 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-socket\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.290323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.285859 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istiod-ca-cert\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.290323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.285969 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-certs\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.290323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.286053 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-envoy\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.351326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.351142 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" event={"ID":"5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4","Type":"ContainerDied","Data":"49f2b784e4607db61b3cacdfa9601423d42fffe825545ded83ab58dadedcc65d"} Oct 31 21:53:46.351326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.351206 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49f2b784e4607db61b3cacdfa9601423d42fffe825545ded83ab58dadedcc65d" Oct 31 21:53:46.351326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.351282 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h" Oct 31 21:53:46.375323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.373714 4294 generic.go:356] "Generic (PLEG): container finished" podID="fff879a9-687a-48dc-b16b-edd4de1c1ae9" containerID="2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254" exitCode=137 Oct 31 21:53:46.375323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.373800 4294 scope.go:122] "RemoveContainer" containerID="2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254" Oct 31 21:53:46.375323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.373949 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7546-35d504" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389148 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istiod-ca-cert\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389237 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-certs\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389332 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-envoy\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389407 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7w6\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-kube-api-access-kd7w6\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389475 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-podinfo\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389506 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-data\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389534 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-credential-socket\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389601 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-token\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.389631 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-socket\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.390110 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-socket\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.390688 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istiod-ca-cert\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.391328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.390920 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-certs\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.396344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.395814 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-data\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.396344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.396106 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-credential-socket\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.448169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.448132 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-envoy\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.448468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.448444 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7w6\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-kube-api-access-kd7w6\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.449163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.449042 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-podinfo\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.450421 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.450388 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-token\") pod \"e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.472487 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.472464 4294 scope.go:122] "RemoveContainer" containerID="2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254" Oct 31 21:53:46.472885 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:53:46.472865 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254\": container with ID starting with 2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254 not found: ID does not exist" containerID="2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254" Oct 31 21:53:46.472942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.472897 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254"} err="failed to get container status \"2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254\": rpc error: code = NotFound desc = could not find container \"2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254\": container with ID starting with 2b3766fbccbee51d380dd611017edae1effa1660d2123d32be28744afe90d254 not found: ID does not exist" Oct 31 21:53:46.475757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.475735 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:46.855329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.855287 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz"] Oct 31 21:53:46.857517 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:53:46.857492 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc39eac6_d68f_4420_87d7_48f4c0fbbe96.slice/crio-9d5366f7429a6ed7ec5d8f1662d404057a785713fe4b40f6e9cb5fe23ab0f096 WatchSource:0}: Error finding container 9d5366f7429a6ed7ec5d8f1662d404057a785713fe4b40f6e9cb5fe23ab0f096: Status 404 returned error can't find the container with id 9d5366f7429a6ed7ec5d8f1662d404057a785713fe4b40f6e9cb5fe23ab0f096 Oct 31 21:53:46.860729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.860690 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:53:46.860803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.860777 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:53:46.860846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:46.860815 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:53:47.383978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:47.383937 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" event={"ID":"fc39eac6-d68f-4420-87d7-48f4c0fbbe96","Type":"ContainerStarted","Data":"158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e"} Oct 31 21:53:47.384184 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:47.384056 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" event={"ID":"fc39eac6-d68f-4420-87d7-48f4c0fbbe96","Type":"ContainerStarted","Data":"9d5366f7429a6ed7ec5d8f1662d404057a785713fe4b40f6e9cb5fe23ab0f096"} Oct 31 21:53:47.430559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:47.430497 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podStartSLOduration=1.4304796880000001 podStartE2EDuration="1.430479688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:53:47.428218607 +0000 UTC m=+2267.553325839" watchObservedRunningTime="2026-10-31 21:53:47.430479688 +0000 UTC m=+2267.555586921" Oct 31 21:53:47.476316 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:47.476274 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:47.481919 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:47.481891 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:48.392931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:48.392898 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:48.393880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:48.393858 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:53:50.260698 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.260655 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt"] Oct 31 21:53:50.264842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.264822 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.282508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.282481 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt"] Oct 31 21:53:50.290528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.290503 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:53:50.290686 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.290667 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:53:50.364107 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.364075 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.364320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.364263 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.364389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.364372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv6f6\" (UniqueName: \"kubernetes.io/projected/6de5d2a0-55f8-49a1-a095-c954e9937dde-kube-api-access-kv6f6\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.465971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.465934 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.466132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.465991 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv6f6\" (UniqueName: \"kubernetes.io/projected/6de5d2a0-55f8-49a1-a095-c954e9937dde-kube-api-access-kv6f6\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.466132 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.466092 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.466627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.466594 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.468485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.468464 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.489970 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.489945 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv6f6\" (UniqueName: \"kubernetes.io/projected/6de5d2a0-55f8-49a1-a095-c954e9937dde-kube-api-access-kv6f6\") pod \"maas-api-revoke-sub-8fa47f95-hx9wt\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.579935 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.579857 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:50.890161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:50.890056 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt"] Oct 31 21:53:50.907481 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:53:50.907450 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de5d2a0_55f8_49a1_a095_c954e9937dde.slice/crio-0eb1a4c927a1e5f15e963204162a482f31f105bc5b11107edc6b173c1d0e0b32 WatchSource:0}: Error finding container 0eb1a4c927a1e5f15e963204162a482f31f105bc5b11107edc6b173c1d0e0b32: Status 404 returned error can't find the container with id 0eb1a4c927a1e5f15e963204162a482f31f105bc5b11107edc6b173c1d0e0b32 Oct 31 21:53:51.414519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:51.414482 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" event={"ID":"6de5d2a0-55f8-49a1-a095-c954e9937dde","Type":"ContainerStarted","Data":"0d7f4ba4351e6f5b4868d9b88e4c0c627eccc21775aba89eb00ce1d5df9895a9"} Oct 31 21:53:51.414881 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:51.414538 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" event={"ID":"6de5d2a0-55f8-49a1-a095-c954e9937dde","Type":"ContainerStarted","Data":"0eb1a4c927a1e5f15e963204162a482f31f105bc5b11107edc6b173c1d0e0b32"} Oct 31 21:53:51.456971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:51.456880 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" podStartSLOduration=1.456866936 podStartE2EDuration="1.456866936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:53:51.45383675 +0000 UTC m=+2271.578943982" watchObservedRunningTime="2026-10-31 21:53:51.456866936 +0000 UTC m=+2271.581974166" Oct 31 21:53:52.422337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.422290 4294 generic.go:356] "Generic (PLEG): container finished" podID="6de5d2a0-55f8-49a1-a095-c954e9937dde" containerID="0d7f4ba4351e6f5b4868d9b88e4c0c627eccc21775aba89eb00ce1d5df9895a9" exitCode=0 Oct 31 21:53:52.422751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.422336 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" event={"ID":"6de5d2a0-55f8-49a1-a095-c954e9937dde","Type":"ContainerDied","Data":"0d7f4ba4351e6f5b4868d9b88e4c0c627eccc21775aba89eb00ce1d5df9895a9"} Oct 31 21:53:52.795331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.795265 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp"] Oct 31 21:53:52.799360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.799343 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:52.818705 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.818679 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp"] Oct 31 21:53:52.911490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.911461 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:52.911637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.911551 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:52.911637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:52.911575 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4khzl\" (UniqueName: \"kubernetes.io/projected/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-kube-api-access-4khzl\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.013518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.013484 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.013676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.013583 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.013676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.013615 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4khzl\" (UniqueName: \"kubernetes.io/projected/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-kube-api-access-4khzl\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.014193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.014176 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.015912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.015893 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.040178 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.040148 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4khzl\" (UniqueName: \"kubernetes.io/projected/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-kube-api-access-4khzl\") pod \"maas-api-revoke-sub-a5eeb6a0-xqtfp\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.114744 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.114663 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:53.425666 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.425636 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp"] Oct 31 21:53:53.427477 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:53:53.427451 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8fa9c0a_2a96_48d1_8a89_58b4c273d8d4.slice/crio-58b44bacf9c147a008ffc21e04eb0078cd7d6c7fb9990f354c0673e463e88c4d WatchSource:0}: Error finding container 58b44bacf9c147a008ffc21e04eb0078cd7d6c7fb9990f354c0673e463e88c4d: Status 404 returned error can't find the container with id 58b44bacf9c147a008ffc21e04eb0078cd7d6c7fb9990f354c0673e463e88c4d Oct 31 21:53:53.597142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.597119 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:53.725213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.725188 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-xks-ca\") pod \"6de5d2a0-55f8-49a1-a095-c954e9937dde\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " Oct 31 21:53:53.725493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.725474 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6de5d2a0-55f8-49a1-a095-c954e9937dde-kube-api-access-kv6f6\" (UniqueName: \"kubernetes.io/projected/6de5d2a0-55f8-49a1-a095-c954e9937dde-kube-api-access-kv6f6\") pod \"6de5d2a0-55f8-49a1-a095-c954e9937dde\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " Oct 31 21:53:53.725610 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.725590 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-openshift-ca\") pod \"6de5d2a0-55f8-49a1-a095-c954e9937dde\" (UID: \"6de5d2a0-55f8-49a1-a095-c954e9937dde\") " Oct 31 21:53:53.725925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.725900 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-openshift-ca" pod "6de5d2a0-55f8-49a1-a095-c954e9937dde" (UID: "6de5d2a0-55f8-49a1-a095-c954e9937dde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:53.726076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.726059 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:53.727127 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.727103 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-xks-ca" pod "6de5d2a0-55f8-49a1-a095-c954e9937dde" (UID: "6de5d2a0-55f8-49a1-a095-c954e9937dde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:53:53.727328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.727293 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de5d2a0-55f8-49a1-a095-c954e9937dde-kube-api-access-kv6f6" pod "6de5d2a0-55f8-49a1-a095-c954e9937dde" (UID: "6de5d2a0-55f8-49a1-a095-c954e9937dde"). InnerVolumeSpecName "kube-api-access-kv6f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:53.827377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.827333 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6de5d2a0-55f8-49a1-a095-c954e9937dde-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:53.827377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:53.827373 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv6f6\" (UniqueName: \"kubernetes.io/projected/6de5d2a0-55f8-49a1-a095-c954e9937dde-kube-api-access-kv6f6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:54.440663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.440632 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" Oct 31 21:53:54.441082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.440635 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt" event={"ID":"6de5d2a0-55f8-49a1-a095-c954e9937dde","Type":"ContainerDied","Data":"0eb1a4c927a1e5f15e963204162a482f31f105bc5b11107edc6b173c1d0e0b32"} Oct 31 21:53:54.441082 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.440794 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb1a4c927a1e5f15e963204162a482f31f105bc5b11107edc6b173c1d0e0b32" Oct 31 21:53:54.442287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.442264 4294 generic.go:356] "Generic (PLEG): container finished" podID="b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" containerID="1534c5ec4e8731d6feec82481416fc111d622dca2308a955e5f9322bdb20c033" exitCode=0 Oct 31 21:53:54.442408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.442331 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" event={"ID":"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4","Type":"ContainerDied","Data":"1534c5ec4e8731d6feec82481416fc111d622dca2308a955e5f9322bdb20c033"} Oct 31 21:53:54.442408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.442380 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" event={"ID":"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4","Type":"ContainerStarted","Data":"58b44bacf9c147a008ffc21e04eb0078cd7d6c7fb9990f354c0673e463e88c4d"} Oct 31 21:53:54.734283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.734256 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z"] Oct 31 21:53:54.734483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.734461 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" containerID="cri-o://ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813" gracePeriod=30 Oct 31 21:53:54.739260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.739233 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r"] Oct 31 21:53:54.739531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:54.739505 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" containerID="cri-o://90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2" gracePeriod=30 Oct 31 21:53:55.600880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.600853 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:55.655224 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.655188 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-xks-ca\") pod \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " Oct 31 21:53:55.655561 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.655534 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-openshift-ca\") pod \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " Oct 31 21:53:55.655696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.655641 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-kube-api-access-4khzl\" (UniqueName: \"kubernetes.io/projected/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-kube-api-access-4khzl\") pod \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\" (UID: \"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4\") " Oct 31 21:53:55.659854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.659823 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-openshift-ca" pod "b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" (UID: "b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:53:55.661860 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.661833 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:55.664954 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.664922 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-xks-ca" pod "b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" (UID: "b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:53:55.672844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.672806 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-kube-api-access-4khzl" pod "b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" (UID: "b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4"). InnerVolumeSpecName "kube-api-access-4khzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:53:55.763256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.763225 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4khzl\" (UniqueName: \"kubernetes.io/projected/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-kube-api-access-4khzl\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:55.763256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:55.763252 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:53:56.457085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:56.457057 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" Oct 31 21:53:56.457262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:56.457050 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp" event={"ID":"b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4","Type":"ContainerDied","Data":"58b44bacf9c147a008ffc21e04eb0078cd7d6c7fb9990f354c0673e463e88c4d"} Oct 31 21:53:56.457262 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:53:56.457176 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b44bacf9c147a008ffc21e04eb0078cd7d6c7fb9990f354c0673e463e88c4d" Oct 31 21:54:00.015121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.015092 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:54:00.033110 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.033085 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:54:00.123178 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123148 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-data\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.123375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123260 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-certs\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.123375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123355 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-podinfo\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.123502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123390 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-credential-socket\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.123502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123426 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-token\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-token\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.123502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123456 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-podinfo\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.123502 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123471 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-data" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.123693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123562 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-certs" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.123733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123706 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-credential-socket" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.123859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123836 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-envoy\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.123987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123864 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istiod-ca-cert\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.123987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123909 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-socket\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.123987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123965 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-credential-socket\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.124207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.123999 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e606df7-cf70-401f-af4c-0639a8bdaff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e606df7-cf70-401f-af4c-0639a8bdaff0-istiod-ca-cert\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.124207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124043 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-kube-api-access-ljnw2\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-kube-api-access-ljnw2\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.124207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124133 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-data\") pod \"3e606df7-cf70-401f-af4c-0639a8bdaff0\" (UID: \"3e606df7-cf70-401f-af4c-0639a8bdaff0\") " Oct 31 21:54:00.124207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124161 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-envoy\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.124207 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124202 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-certs\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.124547 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124248 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istiod-ca-cert" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:00.124547 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124279 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-socket\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.124801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124767 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-token\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-token\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.124873 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.124812 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-kube-api-access-f77b4\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-kube-api-access-f77b4\") pod \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\" (UID: \"3b7ffa0f-8116-4c06-a302-cfd9071d85bf\") " Oct 31 21:54:00.125037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125011 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-certs" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.125221 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125197 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-socket" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.125737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125714 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.126413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125740 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.126413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125753 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.126413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125764 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.126413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125776 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.126413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125784 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.126413 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.125981 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-envoy" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.126739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.126430 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-credential-socket" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.126739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.126576 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-podinfo" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:00.126739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.126589 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-socket" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.126888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.126791 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-data" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.126945 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.126911 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e606df7-cf70-401f-af4c-0639a8bdaff0-istiod-ca-cert" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:00.127017 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.126999 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-envoy" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:00.127358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.127333 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-token" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:00.127918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.127888 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-token" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:00.128454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.128429 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-kube-api-access-f77b4" pod "3b7ffa0f-8116-4c06-a302-cfd9071d85bf" (UID: "3b7ffa0f-8116-4c06-a302-cfd9071d85bf"). InnerVolumeSpecName "kube-api-access-f77b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:00.129007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.128967 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-podinfo" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:00.129330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.129294 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-kube-api-access-ljnw2" pod "3e606df7-cf70-401f-af4c-0639a8bdaff0" (UID: "3e606df7-cf70-401f-af4c-0639a8bdaff0"). InnerVolumeSpecName "kube-api-access-ljnw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232292 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232352 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f77b4\" (UniqueName: \"kubernetes.io/projected/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-kube-api-access-f77b4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232367 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232380 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232396 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232409 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232423 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232436 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232451 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e606df7-cf70-401f-af4c-0639a8bdaff0-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232464 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnw2\" (UniqueName: \"kubernetes.io/projected/3e606df7-cf70-401f-af4c-0639a8bdaff0-kube-api-access-ljnw2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232478 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e606df7-cf70-401f-af4c-0639a8bdaff0-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.232660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.232492 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7ffa0f-8116-4c06-a302-cfd9071d85bf-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:00.488014 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.487979 4294 generic.go:356] "Generic (PLEG): container finished" podID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerID="ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813" exitCode=0 Oct 31 21:54:00.488191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.488031 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" Oct 31 21:54:00.488191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.488059 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" event={"ID":"3e606df7-cf70-401f-af4c-0639a8bdaff0","Type":"ContainerDied","Data":"ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813"} Oct 31 21:54:00.488191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.488111 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z" event={"ID":"3e606df7-cf70-401f-af4c-0639a8bdaff0","Type":"ContainerDied","Data":"2ced035bf46c09528d66e22b8275f2fb6adfaa27acec4fcc5fbcec920f124675"} Oct 31 21:54:00.488191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.488141 4294 scope.go:122] "RemoveContainer" containerID="ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813" Oct 31 21:54:00.489594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.489574 4294 generic.go:356] "Generic (PLEG): container finished" podID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerID="90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2" exitCode=0 Oct 31 21:54:00.489703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.489603 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" event={"ID":"3b7ffa0f-8116-4c06-a302-cfd9071d85bf","Type":"ContainerDied","Data":"90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2"} Oct 31 21:54:00.489703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.489615 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" Oct 31 21:54:00.489703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.489645 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r" event={"ID":"3b7ffa0f-8116-4c06-a302-cfd9071d85bf","Type":"ContainerDied","Data":"b74df6d91af0f0f30264addd6b87898491b359e541a2ea75d49cdc64d074e71d"} Oct 31 21:54:00.499861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.499844 4294 scope.go:122] "RemoveContainer" containerID="ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813" Oct 31 21:54:00.500107 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:54:00.500089 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813\": container with ID starting with ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813 not found: ID does not exist" containerID="ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813" Oct 31 21:54:00.500169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.500113 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813"} err="failed to get container status \"ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813\": rpc error: code = NotFound desc = could not find container \"ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813\": container with ID starting with ef218124b56f0e899312a35a4eadc8eb76ba4a63a82e46c0230136275c420813 not found: ID does not exist" Oct 31 21:54:00.500169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.500128 4294 scope.go:122] "RemoveContainer" containerID="90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2" Oct 31 21:54:00.509114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.509094 4294 scope.go:122] "RemoveContainer" containerID="90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2" Oct 31 21:54:00.509365 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:54:00.509348 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2\": container with ID starting with 90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2 not found: ID does not exist" containerID="90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2" Oct 31 21:54:00.509431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.509371 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2"} err="failed to get container status \"90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2\": rpc error: code = NotFound desc = could not find container \"90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2\": container with ID starting with 90a570d679ba32624d15250c4b3d280e2f25a28b12c17ca029e4d52ea518e9f2 not found: ID does not exist" Oct 31 21:54:00.541635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.541611 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z"] Oct 31 21:54:00.553652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.553625 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-zt98z"] Oct 31 21:54:00.583309 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.583279 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r"] Oct 31 21:54:00.590832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:00.590802 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-ckh7r"] Oct 31 21:54:01.098429 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098355 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5"] Oct 31 21:54:01.098890 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098874 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" Oct 31 21:54:01.098941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098892 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" Oct 31 21:54:01.098941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098907 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6de5d2a0-55f8-49a1-a095-c954e9937dde" containerName="revoke-keys" Oct 31 21:54:01.098941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098913 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6de5d2a0-55f8-49a1-a095-c954e9937dde" containerName="revoke-keys" Oct 31 21:54:01.098941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098929 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" Oct 31 21:54:01.098941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098933 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" Oct 31 21:54:01.099091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098956 4294 cpu_manager.go:414] "containerMap: removing container" podUID="b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" containerName="revoke-keys" Oct 31 21:54:01.099091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.098961 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" containerName="revoke-keys" Oct 31 21:54:01.099091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.099051 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" containerName="istio-proxy" Oct 31 21:54:01.099091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.099058 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6de5d2a0-55f8-49a1-a095-c954e9937dde" containerName="revoke-keys" Oct 31 21:54:01.099091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.099067 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" containerName="istio-proxy" Oct 31 21:54:01.099091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.099075 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" containerName="revoke-keys" Oct 31 21:54:01.104245 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.104229 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.122570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.122546 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-c34cbd-serving-cert\"" Oct 31 21:54:01.126992 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.126968 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5"] Oct 31 21:54:01.244018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.243983 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pbh\" (UniqueName: \"kubernetes.io/projected/7c46007a-1a41-45b4-b713-96d505118ec4-kube-api-access-b4pbh\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.244018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.244018 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c46007a-1a41-45b4-b713-96d505118ec4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.244245 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.244056 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-metrics-certs\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.244245 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.244142 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.309822 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:54:01.309785 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:54:01.345364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.345332 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pbh\" (UniqueName: \"kubernetes.io/projected/7c46007a-1a41-45b4-b713-96d505118ec4-kube-api-access-b4pbh\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.345364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.345367 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c46007a-1a41-45b4-b713-96d505118ec4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.345602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.345399 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-metrics-certs\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.345602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.345432 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.346040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.346014 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c46007a-1a41-45b4-b713-96d505118ec4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.347771 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.347751 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.348035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.348013 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-metrics-certs\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.388348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.385554 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pbh\" (UniqueName: \"kubernetes.io/projected/7c46007a-1a41-45b4-b713-96d505118ec4-kube-api-access-b4pbh\") pod \"maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.417251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.417221 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:01.670278 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.670201 4294 scope.go:122] "RemoveContainer" containerID="44bd7714323e7f191364c98a14622692def177c139c7a558c9b403ed5854c9e9" Oct 31 21:54:01.680241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.680224 4294 scope.go:122] "RemoveContainer" containerID="bc0a6ce1e17c3c079a4feb5700873efcc94d7adc4699d757c91dc0d08c52e069" Oct 31 21:54:01.688858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.688828 4294 scope.go:122] "RemoveContainer" containerID="8ec36ca39094644d3624a8a43fbf0aa5c7ef8457dd969dbe6f5d88f97eeea5a3" Oct 31 21:54:01.697182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.697162 4294 scope.go:122] "RemoveContainer" containerID="dcafefc2868360a97469dd0636b61db461ac465f798ce44043629ac8f67e378a" Oct 31 21:54:01.709176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.709154 4294 scope.go:122] "RemoveContainer" containerID="937726b9f29573157dc89c3dda9d6ca158e130802896a20c99cc955b73f45731" Oct 31 21:54:01.717266 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.717251 4294 scope.go:122] "RemoveContainer" containerID="27dd9206599f6bb1530d3fe0a7925455cb5c9d8b12ebe8eb5a0c7af5b655369c" Oct 31 21:54:01.725651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.725628 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5"] Oct 31 21:54:01.727144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.727124 4294 scope.go:122] "RemoveContainer" containerID="fd3d9e696a16096c4c5f39858863a2542d73adf66733509805b5fb5795efdba0" Oct 31 21:54:01.728820 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:54:01.728798 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c46007a_1a41_45b4_b713_96d505118ec4.slice/crio-50aeb03a563b403153c20204182cfa16f13aa866af7fe0d32beb3360fb451fe9 WatchSource:0}: Error finding container 50aeb03a563b403153c20204182cfa16f13aa866af7fe0d32beb3360fb451fe9: Status 404 returned error can't find the container with id 50aeb03a563b403153c20204182cfa16f13aa866af7fe0d32beb3360fb451fe9 Oct 31 21:54:01.730104 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.730089 4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 21:54:01.735441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.735422 4294 scope.go:122] "RemoveContainer" containerID="c67bb9733a8a97e5454258009e1e71e9b29fbaa53dc98cef8a1bbfd3d9382c4e" Oct 31 21:54:01.989779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.989750 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b7ffa0f-8116-4c06-a302-cfd9071d85bf" path="/var/lib/kubelet/pods/3b7ffa0f-8116-4c06-a302-cfd9071d85bf/volumes" Oct 31 21:54:01.990341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:01.990326 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e606df7-cf70-401f-af4c-0639a8bdaff0" path="/var/lib/kubelet/pods/3e606df7-cf70-401f-af4c-0639a8bdaff0/volumes" Oct 31 21:54:02.508344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.508280 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" event={"ID":"7c46007a-1a41-45b4-b713-96d505118ec4","Type":"ContainerStarted","Data":"6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d"} Oct 31 21:54:02.508729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.508353 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" event={"ID":"7c46007a-1a41-45b4-b713-96d505118ec4","Type":"ContainerStarted","Data":"50aeb03a563b403153c20204182cfa16f13aa866af7fe0d32beb3360fb451fe9"} Oct 31 21:54:02.508729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.508474 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:02.560286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.560229 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" podStartSLOduration=1.157562546 podStartE2EDuration="1.560216783s" totalImagesPullingTime="402.654237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:02.557370697 +0000 UTC m=+2282.682477921" watchObservedRunningTime="2026-10-31 21:54:02.560216783 +0000 UTC m=+2282.685324014" Oct 31 21:54:02.644165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.644134 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl"] Oct 31 21:54:02.657999 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.657971 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6"] Oct 31 21:54:02.662153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.662129 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.702012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.701963 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-44405d-openshift-default-dockercfg-vjr7p\"" Oct 31 21:54:02.706342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.705662 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6"] Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766059 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-credential-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766133 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-token\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766217 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f55d66-e39e-46ac-8b3c-23d40268f95b-istiod-ca-cert\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766253 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-data\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766358 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhpkp\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-kube-api-access-vhpkp\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766418 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766469 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-certs\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766553 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-envoy\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.807400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.766588 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-podinfo\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.867644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.867605 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-envoy\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.867816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.867664 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-podinfo\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.867816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.867757 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-credential-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.867816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.867810 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-token\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.867989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.867889 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f55d66-e39e-46ac-8b3c-23d40268f95b-istiod-ca-cert\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.867989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.867921 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-data\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.868095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.868006 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhpkp\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-kube-api-access-vhpkp\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.868095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.868084 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.868250 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.868137 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-certs\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.869009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.868679 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-certs\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.869009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.868706 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-credential-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.869009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.868857 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f55d66-e39e-46ac-8b3c-23d40268f95b-istiod-ca-cert\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.869009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.868943 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-data\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.869339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.869165 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-socket\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.870616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.870590 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-podinfo\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.871754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.871727 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-envoy\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.891966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.891937 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-token\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.892212 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.892194 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhpkp\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-kube-api-access-vhpkp\") pod \"e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:02.980676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:02.980621 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:03.095845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.095758 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl"] Oct 31 21:54:03.096012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.095985 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" containerID="cri-o://d55bd8882eabc327dba490dc10ddec3db0032dc678f14dbfa7201fc6803fb508" gracePeriod=30 Oct 31 21:54:03.306018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.305980 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6"] Oct 31 21:54:03.306671 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:54:03.306639 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f55d66_e39e_46ac_8b3c_23d40268f95b.slice/crio-d99cd5a0b3d2769aa93eb658801e18cb2927386e6185cdbfc59efa7e6983d0b4 WatchSource:0}: Error finding container d99cd5a0b3d2769aa93eb658801e18cb2927386e6185cdbfc59efa7e6983d0b4: Status 404 returned error can't find the container with id d99cd5a0b3d2769aa93eb658801e18cb2927386e6185cdbfc59efa7e6983d0b4 Oct 31 21:54:03.310565 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.310523 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:03.310679 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.310661 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:03.310765 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.310750 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:03.523131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.523075 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" event={"ID":"21f55d66-e39e-46ac-8b3c-23d40268f95b","Type":"ContainerStarted","Data":"3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2"} Oct 31 21:54:03.523571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.523152 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" event={"ID":"21f55d66-e39e-46ac-8b3c-23d40268f95b","Type":"ContainerStarted","Data":"d99cd5a0b3d2769aa93eb658801e18cb2927386e6185cdbfc59efa7e6983d0b4"} Oct 31 21:54:03.570700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.570649 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podStartSLOduration=1.570634715 podStartE2EDuration="1.570634715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:03.568024925 +0000 UTC m=+2283.693132159" watchObservedRunningTime="2026-10-31 21:54:03.570634715 +0000 UTC m=+2283.695741946" Oct 31 21:54:03.993744 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:03.993712 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:04.997242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:04.997180 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 21:54:04.997773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:04.997258 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 21:54:05.994327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:05.989405 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 21:54:05.994327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:05.989479 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 21:54:06.984326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:06.981678 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 21:54:06.984326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:06.981739 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 21:54:07.983369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:07.981038 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 21:54:07.983369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:07.981093 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 21:54:08.040654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.039704 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb"] Oct 31 21:54:08.046580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.045591 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.065339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.063405 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-696f9a-openshift-default-dockercfg-d6zn7\"" Oct 31 21:54:08.120410 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.120376 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb"] Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179207 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-data\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179277 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-credential-socket\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179330 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45584a2a-6fd4-45fe-8b86-6add93774c95-istiod-ca-cert\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179364 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-envoy\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179390 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-token\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179440 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-podinfo\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179516 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-certs\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179560 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcd7x\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-kube-api-access-dcd7x\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.179960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.179699 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-socket\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289136 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-socket\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289279 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-data\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289338 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-credential-socket\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289366 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45584a2a-6fd4-45fe-8b86-6add93774c95-istiod-ca-cert\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289399 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-envoy\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289428 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-token\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289476 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-podinfo\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289553 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-certs\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.289596 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcd7x\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-kube-api-access-dcd7x\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.290474 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-socket\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.290720 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-data\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.291331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.290956 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-credential-socket\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.306688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.306622 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45584a2a-6fd4-45fe-8b86-6add93774c95-istiod-ca-cert\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.307710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.307661 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-certs\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.349327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.333115 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcd7x\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-kube-api-access-dcd7x\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.349327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.345461 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-token\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.441068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.440663 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-envoy\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.441068 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.440855 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-podinfo\") pod \"e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.559079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.557033 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 21:54:08.616936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.607679 4294 generic.go:356] "Generic (PLEG): container finished" podID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerID="d55bd8882eabc327dba490dc10ddec3db0032dc678f14dbfa7201fc6803fb508" exitCode=0 Oct 31 21:54:08.616936 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.607791 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" event={"ID":"362cb142-fbaf-4000-8fbc-be3f42e9a3cb","Type":"ContainerDied","Data":"d55bd8882eabc327dba490dc10ddec3db0032dc678f14dbfa7201fc6803fb508"} Oct 31 21:54:08.653852 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.653793 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:54:08.662962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.662931 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:08.849730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.847258 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-envoy\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.850439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.850413 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-data\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.850618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.850595 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-credential-socket\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.850800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.850782 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-podinfo\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.851376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.851035 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-token\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-token\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.851376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.851106 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-socket\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.851376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.851147 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-kube-api-access-xkbfj\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-kube-api-access-xkbfj\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.851376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.851179 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istiod-ca-cert\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.851376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.851227 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-certs\") pod \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\" (UID: \"362cb142-fbaf-4000-8fbc-be3f42e9a3cb\") " Oct 31 21:54:08.855073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.852159 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-certs" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:08.855073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.852398 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-data" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:08.855073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.852582 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-credential-socket" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:08.865601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.864075 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-socket" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:08.867684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.867629 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istiod-ca-cert" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:08.918607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.918501 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-kube-api-access-xkbfj" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "kube-api-access-xkbfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:08.918886 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.918845 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-token" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:08.935441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.931505 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-envoy" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:08.935441 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.933021 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-podinfo" pod "362cb142-fbaf-4000-8fbc-be3f42e9a3cb" (UID: "362cb142-fbaf-4000-8fbc-be3f42e9a3cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:08.956734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956700 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956734 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956733 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956749 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956761 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkbfj\" (UniqueName: \"kubernetes.io/projected/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-kube-api-access-xkbfj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956774 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956788 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956800 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956812 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.956941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.956824 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/362cb142-fbaf-4000-8fbc-be3f42e9a3cb-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:08.989768 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.987665 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 31 21:54:08.989768 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:08.987724 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 31 21:54:09.101718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.100084 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb"] Oct 31 21:54:09.107495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.106997 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:09.107495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.107059 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:09.107495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.107098 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:09.170242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.170213 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l"] Oct 31 21:54:09.171080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.171054 4294 cpu_manager.go:414] "containerMap: removing container" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" Oct 31 21:54:09.171194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.171081 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" Oct 31 21:54:09.172347 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.171355 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" containerName="istio-proxy" Oct 31 21:54:09.196584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.191039 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.217540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.204027 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l"] Oct 31 21:54:09.217540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.206961 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:54:09.217540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.207015 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:54:09.261692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.261663 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.261966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.261940 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k74dw\" (UniqueName: \"kubernetes.io/projected/85479044-fab2-4413-a3f5-12d458699ec0-kube-api-access-k74dw\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.262044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.262002 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.362946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.362852 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k74dw\" (UniqueName: \"kubernetes.io/projected/85479044-fab2-4413-a3f5-12d458699ec0-kube-api-access-k74dw\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.362946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.362906 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.363183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.362957 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.363947 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.363921 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.365998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.365970 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.393119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.393086 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k74dw\" (UniqueName: \"kubernetes.io/projected/85479044-fab2-4413-a3f5-12d458699ec0-kube-api-access-k74dw\") pod \"maas-api-revoke-sub-720a0285-g2j7l\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.532286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.532243 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:09.627162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.626293 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" event={"ID":"362cb142-fbaf-4000-8fbc-be3f42e9a3cb","Type":"ContainerDied","Data":"2b9f24102144c6de82d37efd70ac95e8f26b561b8b7d23613d0804b958ab46f2"} Oct 31 21:54:09.627162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.626386 4294 scope.go:122] "RemoveContainer" containerID="d55bd8882eabc327dba490dc10ddec3db0032dc678f14dbfa7201fc6803fb508" Oct 31 21:54:09.627162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.626562 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl" Oct 31 21:54:09.642333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.639845 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" event={"ID":"45584a2a-6fd4-45fe-8b86-6add93774c95","Type":"ContainerStarted","Data":"9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41"} Oct 31 21:54:09.642333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.639979 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" event={"ID":"45584a2a-6fd4-45fe-8b86-6add93774c95","Type":"ContainerStarted","Data":"5594045b8645dec1924b016cc0d8359e102dbc81f140a52ecb39694eac6e1e52"} Oct 31 21:54:09.666619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.663861 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:09.718669 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.718609 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" podStartSLOduration=2.718590484 podStartE2EDuration="2.718590484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:09.715375333 +0000 UTC m=+2289.840482568" watchObservedRunningTime="2026-10-31 21:54:09.718590484 +0000 UTC m=+2289.843697715" Oct 31 21:54:09.766539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.766499 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl"] Oct 31 21:54:09.778447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.778414 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-2rnbl"] Oct 31 21:54:09.868109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.868077 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l"] Oct 31 21:54:09.872127 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:54:09.872095 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85479044_fab2_4413_a3f5_12d458699ec0.slice/crio-1bae713377d238bf0a17c806cd1e52223bc577ea64bd27d8a4e94920cfe319af WatchSource:0}: Error finding container 1bae713377d238bf0a17c806cd1e52223bc577ea64bd27d8a4e94920cfe319af: Status 404 returned error can't find the container with id 1bae713377d238bf0a17c806cd1e52223bc577ea64bd27d8a4e94920cfe319af Oct 31 21:54:09.985867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.985836 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="362cb142-fbaf-4000-8fbc-be3f42e9a3cb" path="/var/lib/kubelet/pods/362cb142-fbaf-4000-8fbc-be3f42e9a3cb/volumes" Oct 31 21:54:09.986471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.986453 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:09.986893 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.986872 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:09.987751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:09.987732 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 21:54:10.664973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:10.664930 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" event={"ID":"85479044-fab2-4413-a3f5-12d458699ec0","Type":"ContainerStarted","Data":"fe2911d88d83e9faa7507d5a9cc19adaa8a766f4ff270edb3efd378c74d80eed"} Oct 31 21:54:10.665495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:10.665031 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" event={"ID":"85479044-fab2-4413-a3f5-12d458699ec0","Type":"ContainerStarted","Data":"1bae713377d238bf0a17c806cd1e52223bc577ea64bd27d8a4e94920cfe319af"} Oct 31 21:54:10.670543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:10.670516 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:10.706697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:10.706642 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" podStartSLOduration=1.706628093 podStartE2EDuration="1.706628093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:10.705097294 +0000 UTC m=+2290.830204525" watchObservedRunningTime="2026-10-31 21:54:10.706628093 +0000 UTC m=+2290.831735324" Oct 31 21:54:11.672823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:11.672787 4294 generic.go:356] "Generic (PLEG): container finished" podID="85479044-fab2-4413-a3f5-12d458699ec0" containerID="fe2911d88d83e9faa7507d5a9cc19adaa8a766f4ff270edb3efd378c74d80eed" exitCode=0 Oct 31 21:54:11.673443 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:11.672860 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" event={"ID":"85479044-fab2-4413-a3f5-12d458699ec0","Type":"ContainerDied","Data":"fe2911d88d83e9faa7507d5a9cc19adaa8a766f4ff270edb3efd378c74d80eed"} Oct 31 21:54:11.673443 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:11.673281 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:11.674149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:11.674129 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:54:12.827975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.827951 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:12.921681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.921652 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-openshift-ca\") pod \"85479044-fab2-4413-a3f5-12d458699ec0\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " Oct 31 21:54:12.921845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.921797 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85479044-fab2-4413-a3f5-12d458699ec0-kube-api-access-k74dw\" (UniqueName: \"kubernetes.io/projected/85479044-fab2-4413-a3f5-12d458699ec0-kube-api-access-k74dw\") pod \"85479044-fab2-4413-a3f5-12d458699ec0\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " Oct 31 21:54:12.921845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.921832 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-xks-ca\") pod \"85479044-fab2-4413-a3f5-12d458699ec0\" (UID: \"85479044-fab2-4413-a3f5-12d458699ec0\") " Oct 31 21:54:12.922044 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.922015 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-openshift-ca" pod "85479044-fab2-4413-a3f5-12d458699ec0" (UID: "85479044-fab2-4413-a3f5-12d458699ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:12.922374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.922351 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:12.923844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.923819 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85479044-fab2-4413-a3f5-12d458699ec0-kube-api-access-k74dw" pod "85479044-fab2-4413-a3f5-12d458699ec0" (UID: "85479044-fab2-4413-a3f5-12d458699ec0"). InnerVolumeSpecName "kube-api-access-k74dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:12.923942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:12.923858 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-xks-ca" pod "85479044-fab2-4413-a3f5-12d458699ec0" (UID: "85479044-fab2-4413-a3f5-12d458699ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:54:13.023477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:13.023448 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k74dw\" (UniqueName: \"kubernetes.io/projected/85479044-fab2-4413-a3f5-12d458699ec0-kube-api-access-k74dw\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:13.023477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:13.023474 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85479044-fab2-4413-a3f5-12d458699ec0-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:13.694535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:13.694497 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" event={"ID":"85479044-fab2-4413-a3f5-12d458699ec0","Type":"ContainerDied","Data":"1bae713377d238bf0a17c806cd1e52223bc577ea64bd27d8a4e94920cfe319af"} Oct 31 21:54:13.694535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:13.694534 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l" Oct 31 21:54:13.694756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:13.694581 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bae713377d238bf0a17c806cd1e52223bc577ea64bd27d8a4e94920cfe319af" Oct 31 21:54:17.686783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:17.686744 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" podUID="4a6be379-1483-4028-8935-4f727891e84f" containerName="main" containerID="cri-o://e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c" gracePeriod=45 Oct 31 21:54:17.957154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:17.957129 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:54:18.001548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.001518 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a6be379-1483-4028-8935-4f727891e84f-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6be379-1483-4028-8935-4f727891e84f-tls-certs\") pod \"4a6be379-1483-4028-8935-4f727891e84f\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " Oct 31 21:54:18.001704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.001574 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-tmp-dir\") pod \"4a6be379-1483-4028-8935-4f727891e84f\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " Oct 31 21:54:18.001752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.001712 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6be379-1483-4028-8935-4f727891e84f-kube-api-access-9c2mh\" (UniqueName: \"kubernetes.io/projected/4a6be379-1483-4028-8935-4f727891e84f-kube-api-access-9c2mh\") pod \"4a6be379-1483-4028-8935-4f727891e84f\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " Oct 31 21:54:18.001752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.001748 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-model-cache\") pod \"4a6be379-1483-4028-8935-4f727891e84f\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " Oct 31 21:54:18.001854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.001766 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-dshm\") pod \"4a6be379-1483-4028-8935-4f727891e84f\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " Oct 31 21:54:18.001854 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.001831 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-tmp-dir" pod "4a6be379-1483-4028-8935-4f727891e84f" (UID: "4a6be379-1483-4028-8935-4f727891e84f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:18.001966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.001842 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-home\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-home\") pod \"4a6be379-1483-4028-8935-4f727891e84f\" (UID: \"4a6be379-1483-4028-8935-4f727891e84f\") " Oct 31 21:54:18.002054 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.002008 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-model-cache" pod "4a6be379-1483-4028-8935-4f727891e84f" (UID: "4a6be379-1483-4028-8935-4f727891e84f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:18.002125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.002063 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-home" pod "4a6be379-1483-4028-8935-4f727891e84f" (UID: "4a6be379-1483-4028-8935-4f727891e84f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:18.002610 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.002589 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:18.002682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.002616 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:18.002682 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.002633 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:18.003907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.003877 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-dshm" pod "4a6be379-1483-4028-8935-4f727891e84f" (UID: "4a6be379-1483-4028-8935-4f727891e84f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:18.004005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.003915 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a6be379-1483-4028-8935-4f727891e84f-tls-certs" pod "4a6be379-1483-4028-8935-4f727891e84f" (UID: "4a6be379-1483-4028-8935-4f727891e84f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:54:18.004005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.003942 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6be379-1483-4028-8935-4f727891e84f-kube-api-access-9c2mh" pod "4a6be379-1483-4028-8935-4f727891e84f" (UID: "4a6be379-1483-4028-8935-4f727891e84f"). InnerVolumeSpecName "kube-api-access-9c2mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:18.103869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.103837 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a6be379-1483-4028-8935-4f727891e84f-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:18.103869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.103867 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c2mh\" (UniqueName: \"kubernetes.io/projected/4a6be379-1483-4028-8935-4f727891e84f-kube-api-access-9c2mh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:18.103869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.103877 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a6be379-1483-4028-8935-4f727891e84f-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:18.727846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.727809 4294 generic.go:356] "Generic (PLEG): container finished" podID="4a6be379-1483-4028-8935-4f727891e84f" containerID="e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c" exitCode=0 Oct 31 21:54:18.728270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.727862 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" Oct 31 21:54:18.728270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.727885 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" event={"ID":"4a6be379-1483-4028-8935-4f727891e84f","Type":"ContainerDied","Data":"e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c"} Oct 31 21:54:18.728270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.727928 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" event={"ID":"4a6be379-1483-4028-8935-4f727891e84f","Type":"ContainerDied","Data":"ceb2890f720c4af25af60814b6c0d474904f4aef00535cb630a283c9ee697bfa"} Oct 31 21:54:18.728270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.727948 4294 scope.go:122] "RemoveContainer" containerID="e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c" Oct 31 21:54:18.739144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.739123 4294 scope.go:122] "RemoveContainer" containerID="e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c" Oct 31 21:54:18.739531 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:54:18.739512 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c\": container with ID starting with e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c not found: ID does not exist" containerID="e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c" Oct 31 21:54:18.739609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.739545 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c"} err="failed to get container status \"e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c\": rpc error: code = NotFound desc = could not find container \"e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c\": container with ID starting with e5b6cdffc17d9f8485caea91e13a948cfdb3850d156d9202d576be9c9ec4563c not found: ID does not exist" Oct 31 21:54:18.781052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.781010 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl"] Oct 31 21:54:18.793006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.792980 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl"] Oct 31 21:54:18.913080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:18.913034 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-b36e62/auth-test-model-b36e62-kserve-7cc785ff7-7jqkl" podUID="4a6be379-1483-4028-8935-4f727891e84f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.166:8000/ready\": context deadline exceeded" Oct 31 21:54:19.578023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.577986 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2"] Oct 31 21:54:19.578588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.578575 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4a6be379-1483-4028-8935-4f727891e84f" containerName="main" Oct 31 21:54:19.578656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.578590 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6be379-1483-4028-8935-4f727891e84f" containerName="main" Oct 31 21:54:19.578656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.578614 4294 cpu_manager.go:414] "containerMap: removing container" podUID="85479044-fab2-4413-a3f5-12d458699ec0" containerName="revoke-keys" Oct 31 21:54:19.578656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.578620 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85479044-fab2-4413-a3f5-12d458699ec0" containerName="revoke-keys" Oct 31 21:54:19.578758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.578726 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6be379-1483-4028-8935-4f727891e84f" containerName="main" Oct 31 21:54:19.578758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.578737 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="85479044-fab2-4413-a3f5-12d458699ec0" containerName="revoke-keys" Oct 31 21:54:19.583434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.583416 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.601486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.601457 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2"] Oct 31 21:54:19.609154 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.609127 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:54:19.609326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.609183 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:54:19.723535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.723504 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.723710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.723675 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.723812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.723790 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6p4\" (UniqueName: \"kubernetes.io/projected/a3927850-0b21-4f67-a006-966a5239f32c-kube-api-access-nf6p4\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.825156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.825114 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.825647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.825257 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.825647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.825358 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6p4\" (UniqueName: \"kubernetes.io/projected/a3927850-0b21-4f67-a006-966a5239f32c-kube-api-access-nf6p4\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.825838 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.825818 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.827922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.827895 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.869415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.869341 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6p4\" (UniqueName: \"kubernetes.io/projected/a3927850-0b21-4f67-a006-966a5239f32c-kube-api-access-nf6p4\") pod \"maas-api-revoke-sub-932716b2-qkjk2\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.897396 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.897363 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:19.990199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:19.990169 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6be379-1483-4028-8935-4f727891e84f" path="/var/lib/kubelet/pods/4a6be379-1483-4028-8935-4f727891e84f/volumes" Oct 31 21:54:20.205438 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:20.205410 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2"] Oct 31 21:54:20.206450 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:54:20.206415 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3927850_0b21_4f67_a006_966a5239f32c.slice/crio-effd286cb45869726a295e1c932cf00e8459f5b3afb66141f4b14a6589fe9e91 WatchSource:0}: Error finding container effd286cb45869726a295e1c932cf00e8459f5b3afb66141f4b14a6589fe9e91: Status 404 returned error can't find the container with id effd286cb45869726a295e1c932cf00e8459f5b3afb66141f4b14a6589fe9e91 Oct 31 21:54:20.745445 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:20.745406 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" event={"ID":"a3927850-0b21-4f67-a006-966a5239f32c","Type":"ContainerStarted","Data":"a5907cbae4f1c7369eb9cdf299b4bc3a8d0b9de8d1aa8a8cf865a69de4edc2ed"} Oct 31 21:54:20.745638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:20.745463 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" event={"ID":"a3927850-0b21-4f67-a006-966a5239f32c","Type":"ContainerStarted","Data":"effd286cb45869726a295e1c932cf00e8459f5b3afb66141f4b14a6589fe9e91"} Oct 31 21:54:20.799912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:20.799850 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" podStartSLOduration=1.799832262 podStartE2EDuration="1.799832262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:20.791704464 +0000 UTC m=+2300.916811696" watchObservedRunningTime="2026-10-31 21:54:20.799832262 +0000 UTC m=+2300.924939494" Oct 31 21:54:21.754569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:21.754534 4294 generic.go:356] "Generic (PLEG): container finished" podID="a3927850-0b21-4f67-a006-966a5239f32c" containerID="a5907cbae4f1c7369eb9cdf299b4bc3a8d0b9de8d1aa8a8cf865a69de4edc2ed" exitCode=0 Oct 31 21:54:21.754980 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:21.754680 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" event={"ID":"a3927850-0b21-4f67-a006-966a5239f32c","Type":"ContainerDied","Data":"a5907cbae4f1c7369eb9cdf299b4bc3a8d0b9de8d1aa8a8cf865a69de4edc2ed"} Oct 31 21:54:22.908802 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:22.908773 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:23.068180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.068098 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3927850-0b21-4f67-a006-966a5239f32c-kube-api-access-nf6p4\" (UniqueName: \"kubernetes.io/projected/a3927850-0b21-4f67-a006-966a5239f32c-kube-api-access-nf6p4\") pod \"a3927850-0b21-4f67-a006-966a5239f32c\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " Oct 31 21:54:23.068344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.068223 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-openshift-ca\") pod \"a3927850-0b21-4f67-a006-966a5239f32c\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " Oct 31 21:54:23.068344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.068251 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-xks-ca\") pod \"a3927850-0b21-4f67-a006-966a5239f32c\" (UID: \"a3927850-0b21-4f67-a006-966a5239f32c\") " Oct 31 21:54:23.068637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.068607 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-openshift-ca" pod "a3927850-0b21-4f67-a006-966a5239f32c" (UID: "a3927850-0b21-4f67-a006-966a5239f32c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:23.068972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.068943 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:23.070291 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.070272 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-xks-ca" pod "a3927850-0b21-4f67-a006-966a5239f32c" (UID: "a3927850-0b21-4f67-a006-966a5239f32c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:54:23.070409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.070386 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3927850-0b21-4f67-a006-966a5239f32c-kube-api-access-nf6p4" pod "a3927850-0b21-4f67-a006-966a5239f32c" (UID: "a3927850-0b21-4f67-a006-966a5239f32c"). InnerVolumeSpecName "kube-api-access-nf6p4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:23.170197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.170167 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3927850-0b21-4f67-a006-966a5239f32c-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:23.170197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.170193 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6p4\" (UniqueName: \"kubernetes.io/projected/a3927850-0b21-4f67-a006-966a5239f32c-kube-api-access-nf6p4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:23.770360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.770329 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" Oct 31 21:54:23.770360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.770350 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2" event={"ID":"a3927850-0b21-4f67-a006-966a5239f32c","Type":"ContainerDied","Data":"effd286cb45869726a295e1c932cf00e8459f5b3afb66141f4b14a6589fe9e91"} Oct 31 21:54:23.770563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:23.770402 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="effd286cb45869726a295e1c932cf00e8459f5b3afb66141f4b14a6589fe9e91" Oct 31 21:54:25.043819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:25.043781 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq"] Oct 31 21:54:25.050774 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:25.050747 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df92b58d-xc5qq"] Oct 31 21:54:25.989434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:25.989400 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d32050fa-63f5-4305-81d2-54d854fafbcf" path="/var/lib/kubelet/pods/d32050fa-63f5-4305-81d2-54d854fafbcf/volumes" Oct 31 21:54:27.279804 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:27.279757 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz"] Oct 31 21:54:31.309962 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:54:31.309925 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:54:37.519535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:37.519496 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q"] Oct 31 21:54:37.519920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:37.519694 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" containerID="cri-o://7e54144c4ad675de1dbe3a36ccfc9719df09931e1de401964bcf74e47a8bf63d" gracePeriod=30 Oct 31 21:54:40.865462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:40.865420 4294 patch_prober.go:29] interesting pod/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 21:54:40.865895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:40.865487 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 21:54:42.301766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.301589 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" podUID="ef2624e7-297b-43b7-89c4-a46a67b5067c" containerName="main" containerID="cri-o://51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb" gracePeriod=45 Oct 31 21:54:42.307138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.307094 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" podUID="ef2624e7-297b-43b7-89c4-a46a67b5067c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.168:8000/ready\": EOF" Oct 31 21:54:42.702984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.702882 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:54:42.713485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.713387 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-home\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-home\") pod \"ef2624e7-297b-43b7-89c4-a46a67b5067c\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " Oct 31 21:54:42.714233 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.714175 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-tmp-dir" pod "ef2624e7-297b-43b7-89c4-a46a67b5067c" (UID: "ef2624e7-297b-43b7-89c4-a46a67b5067c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:42.714471 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.714436 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-home" pod "ef2624e7-297b-43b7-89c4-a46a67b5067c" (UID: "ef2624e7-297b-43b7-89c4-a46a67b5067c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:42.714670 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.714608 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-tmp-dir\") pod \"ef2624e7-297b-43b7-89c4-a46a67b5067c\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " Oct 31 21:54:42.715162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.714822 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-model-cache\") pod \"ef2624e7-297b-43b7-89c4-a46a67b5067c\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " Oct 31 21:54:42.715162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.715131 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef2624e7-297b-43b7-89c4-a46a67b5067c-kube-api-access-j8stn\" (UniqueName: \"kubernetes.io/projected/ef2624e7-297b-43b7-89c4-a46a67b5067c-kube-api-access-j8stn\") pod \"ef2624e7-297b-43b7-89c4-a46a67b5067c\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " Oct 31 21:54:42.715859 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.715056 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-model-cache" pod "ef2624e7-297b-43b7-89c4-a46a67b5067c" (UID: "ef2624e7-297b-43b7-89c4-a46a67b5067c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:42.716191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.716130 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-dshm\") pod \"ef2624e7-297b-43b7-89c4-a46a67b5067c\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " Oct 31 21:54:42.719993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.719905 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef2624e7-297b-43b7-89c4-a46a67b5067c-tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2624e7-297b-43b7-89c4-a46a67b5067c-tls-certs\") pod \"ef2624e7-297b-43b7-89c4-a46a67b5067c\" (UID: \"ef2624e7-297b-43b7-89c4-a46a67b5067c\") " Oct 31 21:54:42.721374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.721273 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:42.721549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.721502 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:42.721549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.721530 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:42.752349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.750859 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef2624e7-297b-43b7-89c4-a46a67b5067c-kube-api-access-j8stn" pod "ef2624e7-297b-43b7-89c4-a46a67b5067c" (UID: "ef2624e7-297b-43b7-89c4-a46a67b5067c"). InnerVolumeSpecName "kube-api-access-j8stn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:42.756911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.756855 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-dshm" pod "ef2624e7-297b-43b7-89c4-a46a67b5067c" (UID: "ef2624e7-297b-43b7-89c4-a46a67b5067c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:42.757240 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.757190 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef2624e7-297b-43b7-89c4-a46a67b5067c-tls-certs" pod "ef2624e7-297b-43b7-89c4-a46a67b5067c" (UID: "ef2624e7-297b-43b7-89c4-a46a67b5067c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:54:42.823214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.823136 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef2624e7-297b-43b7-89c4-a46a67b5067c-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:42.823214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.823172 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef2624e7-297b-43b7-89c4-a46a67b5067c-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:42.823214 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.823189 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8stn\" (UniqueName: \"kubernetes.io/projected/ef2624e7-297b-43b7-89c4-a46a67b5067c-kube-api-access-j8stn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:42.931608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.931529 4294 generic.go:356] "Generic (PLEG): container finished" podID="ef2624e7-297b-43b7-89c4-a46a67b5067c" containerID="51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb" exitCode=0 Oct 31 21:54:42.931608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.931570 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" Oct 31 21:54:42.931819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.931607 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" event={"ID":"ef2624e7-297b-43b7-89c4-a46a67b5067c","Type":"ContainerDied","Data":"51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb"} Oct 31 21:54:42.931819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.931648 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz" event={"ID":"ef2624e7-297b-43b7-89c4-a46a67b5067c","Type":"ContainerDied","Data":"70f91a01c4b57ceb3367e343548f8399e3673033e29e437f5a10b876fc0403d8"} Oct 31 21:54:42.931819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.931666 4294 scope.go:122] "RemoveContainer" containerID="51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb" Oct 31 21:54:42.933792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.933770 4294 generic.go:356] "Generic (PLEG): container finished" podID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerID="7e54144c4ad675de1dbe3a36ccfc9719df09931e1de401964bcf74e47a8bf63d" exitCode=0 Oct 31 21:54:42.933883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.933843 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" event={"ID":"e5f865c1-e469-4ab4-8dfd-4130094d228f","Type":"ContainerDied","Data":"7e54144c4ad675de1dbe3a36ccfc9719df09931e1de401964bcf74e47a8bf63d"} Oct 31 21:54:42.944449 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.944431 4294 scope.go:122] "RemoveContainer" containerID="51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb" Oct 31 21:54:42.944763 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:54:42.944743 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb\": container with ID starting with 51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb not found: ID does not exist" containerID="51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb" Oct 31 21:54:42.944836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:42.944776 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb"} err="failed to get container status \"51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb\": rpc error: code = NotFound desc = could not find container \"51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb\": container with ID starting with 51d210316e87d8b13bf24b71ac222aefb5debed6944623a61c4f81a6f9b22ebb not found: ID does not exist" Oct 31 21:54:43.066355 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.063345 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz"] Oct 31 21:54:43.100392 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.100028 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-6b5dab/auth-test-model-6b5dab-kserve-8fbd5d549-6dbgz"] Oct 31 21:54:43.341287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.341264 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:54:43.346020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.345999 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f865c1-e469-4ab4-8dfd-4130094d228f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f865c1-e469-4ab4-8dfd-4130094d228f-istiod-ca-cert\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346082 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-socket\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346106 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-data\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346122 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-envoy\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346164 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-token\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-token\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346243 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-credential-socket\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346379 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-podinfo\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346392 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-socket" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:43.346461 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346411 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-certs\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346469 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-data" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:43.346701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346479 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-kube-api-access-968nf\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-kube-api-access-968nf\") pod \"e5f865c1-e469-4ab4-8dfd-4130094d228f\" (UID: \"e5f865c1-e469-4ab4-8dfd-4130094d228f\") " Oct 31 21:54:43.346701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346533 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f865c1-e469-4ab4-8dfd-4130094d228f-istiod-ca-cert" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:54:43.346701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346583 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-credential-socket" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:43.346701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.346686 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-certs" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:43.347041 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.347024 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.347125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.347042 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.347125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.347052 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5f865c1-e469-4ab4-8dfd-4130094d228f-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.347125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.347064 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.347125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.347077 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.350439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.350413 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-podinfo" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:54:43.350439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.350422 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-envoy" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:54:43.350608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.350446 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-kube-api-access-968nf" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "kube-api-access-968nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:43.350608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.350474 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-token" pod "e5f865c1-e469-4ab4-8dfd-4130094d228f" (UID: "e5f865c1-e469-4ab4-8dfd-4130094d228f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:54:43.448416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.448389 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.448416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.448414 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.448416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.448426 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5f865c1-e469-4ab4-8dfd-4130094d228f-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.448664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.448435 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-968nf\" (UniqueName: \"kubernetes.io/projected/e5f865c1-e469-4ab4-8dfd-4130094d228f-kube-api-access-968nf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:54:43.942800 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.942770 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" Oct 31 21:54:43.942974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.942793 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q" event={"ID":"e5f865c1-e469-4ab4-8dfd-4130094d228f","Type":"ContainerDied","Data":"4bd79aa0d3e3c7137e651cd14dc517ac1e63019b70b1a9f88d2fc84e9872ecda"} Oct 31 21:54:43.942974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.942876 4294 scope.go:122] "RemoveContainer" containerID="7e54144c4ad675de1dbe3a36ccfc9719df09931e1de401964bcf74e47a8bf63d" Oct 31 21:54:43.987912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:43.987871 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef2624e7-297b-43b7-89c4-a46a67b5067c" path="/var/lib/kubelet/pods/ef2624e7-297b-43b7-89c4-a46a67b5067c/volumes" Oct 31 21:54:44.002125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:44.002092 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q"] Oct 31 21:54:44.014785 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:44.014760 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c16684-openshift-default-77c759c989-wrm6q"] Oct 31 21:54:44.647491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:44.647459 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw"] Oct 31 21:54:45.743973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.743891 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc"] Oct 31 21:54:45.745427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745396 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a3927850-0b21-4f67-a006-966a5239f32c" containerName="revoke-keys" Oct 31 21:54:45.745427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745426 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3927850-0b21-4f67-a006-966a5239f32c" containerName="revoke-keys" Oct 31 21:54:45.745641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745448 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" Oct 31 21:54:45.745641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745457 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" Oct 31 21:54:45.745641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745498 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ef2624e7-297b-43b7-89c4-a46a67b5067c" containerName="main" Oct 31 21:54:45.745641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745506 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef2624e7-297b-43b7-89c4-a46a67b5067c" containerName="main" Oct 31 21:54:45.745870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745644 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" containerName="istio-proxy" Oct 31 21:54:45.745870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745665 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef2624e7-297b-43b7-89c4-a46a67b5067c" containerName="main" Oct 31 21:54:45.745870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.745681 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3927850-0b21-4f67-a006-966a5239f32c" containerName="revoke-keys" Oct 31 21:54:45.755268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.751899 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.774339 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.774280 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc"] Oct 31 21:54:45.785361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.780571 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-f1128d-openshift-default-dockercfg-5tdnz\"" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-podinfo\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787481 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4611dd89-b726-40e0-a371-9ca273fb0b8c-istiod-ca-cert\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787542 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-token\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787570 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-data\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787584 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-envoy\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787624 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-credential-socket\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787654 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-socket\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787671 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-certs\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.789337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.787702 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pp6v\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-kube-api-access-2pp6v\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.889492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.889446 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4611dd89-b726-40e0-a371-9ca273fb0b8c-istiod-ca-cert\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.889881 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.889846 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-token\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.890057 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.890033 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-data\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.890186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.890170 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-envoy\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.890371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.890355 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-credential-socket\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.890530 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.890516 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-socket\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.890668 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.890652 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-certs\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.890814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.890799 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pp6v\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-kube-api-access-2pp6v\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.890949 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.890934 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-podinfo\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.897662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.892085 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4611dd89-b726-40e0-a371-9ca273fb0b8c-istiod-ca-cert\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.899903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.899546 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-data\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.899903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.899710 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-credential-socket\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.907328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.902438 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-envoy\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.907328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.903015 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-socket\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.916168 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.915375 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-certs\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:45.972814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:45.972778 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-podinfo\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:46.028370 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.017280 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-token\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:46.045353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.031611 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f865c1-e469-4ab4-8dfd-4130094d228f" path="/var/lib/kubelet/pods/e5f865c1-e469-4ab4-8dfd-4130094d228f/volumes" Oct 31 21:54:46.054314 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.054229 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pp6v\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-kube-api-access-2pp6v\") pod \"e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:46.094319 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.094275 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:46.455507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.455418 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc"] Oct 31 21:54:46.460442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.460389 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:46.460590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.460528 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:46.460656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:46.460620 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:47.044526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:47.044478 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" event={"ID":"4611dd89-b726-40e0-a371-9ca273fb0b8c","Type":"ContainerStarted","Data":"e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da"} Oct 31 21:54:47.044989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:47.044545 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" event={"ID":"4611dd89-b726-40e0-a371-9ca273fb0b8c","Type":"ContainerStarted","Data":"37a638b6e677a5f24dafeecd0db01462bb96ded585aedca41cb08d6f9ac7ebc0"} Oct 31 21:54:47.097926 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:47.097889 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:47.098877 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:47.098823 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podStartSLOduration=2.09880762 podStartE2EDuration="2.09880762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:47.097713197 +0000 UTC m=+2327.222820455" watchObservedRunningTime="2026-10-31 21:54:47.09880762 +0000 UTC m=+2327.223914854" Oct 31 21:54:47.105374 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:47.105345 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:47.158001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:47.157427 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:54:47.158001 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:47.157478 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:54:48.053098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.053025 4294 generic.go:356] "Generic (PLEG): container finished" podID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerID="4309e06a403166918045e0265f431e08c7791ecf4c98f85bad798b8d1846c8ba" exitCode=0 Oct 31 21:54:48.053590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.053095 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" event={"ID":"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab","Type":"ContainerDied","Data":"4309e06a403166918045e0265f431e08c7791ecf4c98f85bad798b8d1846c8ba"} Oct 31 21:54:48.053590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.053484 4294 scope.go:122] "RemoveContainer" containerID="4309e06a403166918045e0265f431e08c7791ecf4c98f85bad798b8d1846c8ba" Oct 31 21:54:48.053590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.053506 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:48.054485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.054467 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:54:48.055040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.055014 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:48.055091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.055079 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:48.055140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:48.055119 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:54:49.062758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:49.062726 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" event={"ID":"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab","Type":"ContainerStarted","Data":"8f00abbafeb515544bff6908092eba8d8cfce091c98e8b570c1435ed71087bd1"} Oct 31 21:54:49.967308 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:49.967270 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:54:49.967483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:49.967431 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 21:54:49.967483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:49.967470 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 21:54:50.966714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:50.966672 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 21:54:50.967148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:50.966771 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 21:54:51.421324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.421279 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl"] Oct 31 21:54:51.424535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.424516 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.443501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.443473 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl"] Oct 31 21:54:51.450869 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.450849 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f1128d\"" Oct 31 21:54:51.451008 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.450910 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f1128d-dockercfg-4q2d5\"" Oct 31 21:54:51.569327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.542190 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnz77\" (UniqueName: \"kubernetes.io/projected/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-kube-api-access-lnz77\") pod \"payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.569327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.542361 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.650590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.641636 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl"] Oct 31 21:54:51.650590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.647574 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:51.656356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.656009 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl"] Oct 31 21:54:51.687373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.670874 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72b79fe-e6df-4e6d-86e3-c9c45119f449-plugins-config-volume\") pod \"payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:51.687373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.670994 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnz77\" (UniqueName: \"kubernetes.io/projected/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-kube-api-access-lnz77\") pod \"payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.687373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.671142 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcnbj\" (UniqueName: \"kubernetes.io/projected/e72b79fe-e6df-4e6d-86e3-c9c45119f449-kube-api-access-mcnbj\") pod \"payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:51.687373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.671197 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.687373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.672347 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.716990 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.716876 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnz77\" (UniqueName: \"kubernetes.io/projected/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-kube-api-access-lnz77\") pod \"payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.736908 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.736888 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:51.803412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.802390 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcnbj\" (UniqueName: \"kubernetes.io/projected/e72b79fe-e6df-4e6d-86e3-c9c45119f449-kube-api-access-mcnbj\") pod \"payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:51.803412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.802581 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72b79fe-e6df-4e6d-86e3-c9c45119f449-plugins-config-volume\") pod \"payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:51.803412 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.803293 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72b79fe-e6df-4e6d-86e3-c9c45119f449-plugins-config-volume\") pod \"payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:51.839320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.837731 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcnbj\" (UniqueName: \"kubernetes.io/projected/e72b79fe-e6df-4e6d-86e3-c9c45119f449-kube-api-access-mcnbj\") pod \"payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:51.967506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.967426 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 21:54:51.967506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:51.967482 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 21:54:52.046315 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:52.046278 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:52.063450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:52.063424 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl"] Oct 31 21:54:52.065695 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:54:52.065667 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfdee3a_e1e6_40b3_9ff3_c3d796cdafd5.slice/crio-58bc079dff831683b7d376106b55164f7cd854f3eb98cb9f4dc246e7baedbdea WatchSource:0}: Error finding container 58bc079dff831683b7d376106b55164f7cd854f3eb98cb9f4dc246e7baedbdea: Status 404 returned error can't find the container with id 58bc079dff831683b7d376106b55164f7cd854f3eb98cb9f4dc246e7baedbdea Oct 31 21:54:52.113895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:52.113856 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" event={"ID":"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5","Type":"ContainerStarted","Data":"58bc079dff831683b7d376106b55164f7cd854f3eb98cb9f4dc246e7baedbdea"} Oct 31 21:54:52.398241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:52.398213 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl"] Oct 31 21:54:52.401286 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:54:52.401259 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72b79fe_e6df_4e6d_86e3_c9c45119f449.slice/crio-427490d7abca4850423122f891ae1be3d1f3f0b7cc451c40b7f4f9f5525220d3 WatchSource:0}: Error finding container 427490d7abca4850423122f891ae1be3d1f3f0b7cc451c40b7f4f9f5525220d3: Status 404 returned error can't find the container with id 427490d7abca4850423122f891ae1be3d1f3f0b7cc451c40b7f4f9f5525220d3 Oct 31 21:54:52.966598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:52.966559 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 21:54:52.966773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:52.966626 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 21:54:53.121789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.121751 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" event={"ID":"e72b79fe-e6df-4e6d-86e3-c9c45119f449","Type":"ContainerStarted","Data":"60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19"} Oct 31 21:54:53.122238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.121814 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:54:53.122238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.121837 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" event={"ID":"e72b79fe-e6df-4e6d-86e3-c9c45119f449","Type":"ContainerStarted","Data":"427490d7abca4850423122f891ae1be3d1f3f0b7cc451c40b7f4f9f5525220d3"} Oct 31 21:54:53.123474 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.123449 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" event={"ID":"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5","Type":"ContainerStarted","Data":"3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10"} Oct 31 21:54:53.123578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.123501 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:54:53.165594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.165546 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" podStartSLOduration=2.165532432 podStartE2EDuration="2.165532432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:53.161397589 +0000 UTC m=+2333.286504819" watchObservedRunningTime="2026-10-31 21:54:53.165532432 +0000 UTC m=+2333.290639680" Oct 31 21:54:53.209193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.209140 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" podStartSLOduration=2.209125054 podStartE2EDuration="2.209125054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:54:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:54:53.205231986 +0000 UTC m=+2333.330339232" watchObservedRunningTime="2026-10-31 21:54:53.209125054 +0000 UTC m=+2333.334232283" Oct 31 21:54:53.967317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.967278 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 31 21:54:53.967499 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:53.967345 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 31 21:54:54.971072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:54.971036 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:54:54.971605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:54.971398 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:54:54.972122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:54.972102 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 21:54:59.694327 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:59.691549 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" podUID="d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" containerName="main" containerID="cri-o://b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1" gracePeriod=45 Oct 31 21:54:59.766052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:54:59.766003 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" podUID="d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.169:8000/ready\": dial tcp 10.134.0.169:8000: connect: connection refused" Oct 31 21:55:00.041740 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.041714 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:55:00.144880 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.144852 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-kube-api-access-mt6lp\" (UniqueName: \"kubernetes.io/projected/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-kube-api-access-mt6lp\") pod \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " Oct 31 21:55:00.145063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.144896 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tls-certs\") pod \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " Oct 31 21:55:00.145063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.144994 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tmp-dir\") pod \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " Oct 31 21:55:00.145158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.145078 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-home\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-home\") pod \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " Oct 31 21:55:00.145215 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.145197 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-model-cache\") pod \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " Oct 31 21:55:00.145349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.145262 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-dshm\") pod \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\" (UID: \"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e\") " Oct 31 21:55:00.145482 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.145459 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tmp-dir" pod "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" (UID: "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:00.145649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.145632 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-home" pod "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" (UID: "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:00.145817 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.145797 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-model-cache" pod "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" (UID: "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:00.146563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.146538 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:00.146563 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.146565 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:00.146738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.146578 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:00.147646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.147612 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tls-certs" pod "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" (UID: "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:55:00.147756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.147693 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-kube-api-access-mt6lp" pod "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" (UID: "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e"). InnerVolumeSpecName "kube-api-access-mt6lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:55:00.149669 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.149643 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-dshm" pod "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" (UID: "d73abf8e-f6b6-438f-abc9-2a8dbdc4754e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:00.200111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.200082 4294 generic.go:356] "Generic (PLEG): container finished" podID="d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" containerID="b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1" exitCode=0 Oct 31 21:55:00.200356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.200133 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" Oct 31 21:55:00.200356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.200171 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" event={"ID":"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e","Type":"ContainerDied","Data":"b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1"} Oct 31 21:55:00.200356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.200223 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw" event={"ID":"d73abf8e-f6b6-438f-abc9-2a8dbdc4754e","Type":"ContainerDied","Data":"c0cb3443daab8b391e89510e3df585fed8946e81892776417bf2e41db6e5c39a"} Oct 31 21:55:00.200356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.200245 4294 scope.go:122] "RemoveContainer" containerID="b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1" Oct 31 21:55:00.210976 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.210958 4294 scope.go:122] "RemoveContainer" containerID="b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1" Oct 31 21:55:00.211230 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:55:00.211214 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1\": container with ID starting with b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1 not found: ID does not exist" containerID="b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1" Oct 31 21:55:00.211297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.211239 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1"} err="failed to get container status \"b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1\": rpc error: code = NotFound desc = could not find container \"b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1\": container with ID starting with b16381d8d1489bb5a550b667a317b44e0084a332b743db98e6cecef9b28090b1 not found: ID does not exist" Oct 31 21:55:00.247856 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.247831 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:00.247856 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.247856 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt6lp\" (UniqueName: \"kubernetes.io/projected/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-kube-api-access-mt6lp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:00.248016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.247867 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:00.252782 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.252758 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw"] Oct 31 21:55:00.257078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:00.257057 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-44405d/auth-test-model-44405d-kserve-548cd65697-hnxsw"] Oct 31 21:55:01.311529 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:55:01.311488 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:55:01.820040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:01.820008 4294 scope.go:122] "RemoveContainer" containerID="6dc99b8a09a28eba945a669648f7fea02fc34613c063b76134be3327a6c90c8c" Oct 31 21:55:01.830783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:01.830757 4294 scope.go:122] "RemoveContainer" containerID="a95c1d9a153a2284039a4079a36719dcaadb776773c0a282233912635110567f" Oct 31 21:55:01.987843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:01.987804 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" path="/var/lib/kubelet/pods/d73abf8e-f6b6-438f-abc9-2a8dbdc4754e/volumes" Oct 31 21:55:02.191462 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.191376 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx"] Oct 31 21:55:02.191656 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.191626 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" containerID="cri-o://3bc173ed94383494ba57a55aca4072e7aaf23317b7f548afb58b921aecd1b916" gracePeriod=30 Oct 31 21:55:02.216966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.216932 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7540-79de5a"] Oct 31 21:55:02.218091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.218056 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" containerName="main" Oct 31 21:55:02.218091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.218087 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" containerName="main" Oct 31 21:55:02.218344 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.218289 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73abf8e-f6b6-438f-abc9-2a8dbdc4754e" containerName="main" Oct 31 21:55:02.230137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.229554 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:02.243619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.242605 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-79de5a"] Oct 31 21:55:02.265995 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.265968 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dsnkq\"" Oct 31 21:55:02.287222 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.287183 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zqqp\" (UniqueName: \"kubernetes.io/projected/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5-kube-api-access-9zqqp\") pod \"test-curl-7540-79de5a\" (UID: \"c82d32b8-6939-4d1f-b5d8-269e4cfcebe5\") " pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:02.389251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.389211 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zqqp\" (UniqueName: \"kubernetes.io/projected/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5-kube-api-access-9zqqp\") pod \"test-curl-7540-79de5a\" (UID: \"c82d32b8-6939-4d1f-b5d8-269e4cfcebe5\") " pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:02.412857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.412820 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zqqp\" (UniqueName: \"kubernetes.io/projected/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5-kube-api-access-9zqqp\") pod \"test-curl-7540-79de5a\" (UID: \"c82d32b8-6939-4d1f-b5d8-269e4cfcebe5\") " pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:02.555732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.555694 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:02.871673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:02.871191 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7540-79de5a"] Oct 31 21:55:03.225146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:03.225058 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-79de5a" event={"ID":"c82d32b8-6939-4d1f-b5d8-269e4cfcebe5","Type":"ContainerStarted","Data":"cdbf0b445b3f1b034777b0173039528b3b7dcff6a81fe953bd4399f8bf9e8626"} Oct 31 21:55:03.225146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:03.225104 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7540-79de5a" event={"ID":"c82d32b8-6939-4d1f-b5d8-269e4cfcebe5","Type":"ContainerStarted","Data":"3d7d10d833223a0e78d6d74a6920c9d18175d7490fb6de5528783f44e95a7211"} Oct 31 21:55:03.265946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:03.265885 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7540-79de5a" podStartSLOduration=1.265870169 podStartE2EDuration="1.265870169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:03.265582764 +0000 UTC m=+2343.390689996" watchObservedRunningTime="2026-10-31 21:55:03.265870169 +0000 UTC m=+2343.390977400" Oct 31 21:55:04.134858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:04.134823 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:55:04.136455 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:04.136420 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:55:04.196637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:04.190858 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7540-79de5a"] Oct 31 21:55:04.294242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:04.294205 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7540-79de5a"] Oct 31 21:55:04.509126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:04.508713 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 21:55:04.509126 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:04.508797 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 21:55:05.303874 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.303790 4294 generic.go:356] "Generic (PLEG): container finished" podID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerID="158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e" exitCode=0 Oct 31 21:55:05.304386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.303869 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" event={"ID":"fc39eac6-d68f-4420-87d7-48f4c0fbbe96","Type":"ContainerDied","Data":"158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e"} Oct 31 21:55:05.304386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.304114 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7540-79de5a" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" containerName="test-curl-7540-79de5a" containerID="cri-o://cdbf0b445b3f1b034777b0173039528b3b7dcff6a81fe953bd4399f8bf9e8626" gracePeriod=2 Oct 31 21:55:05.304841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.304507 4294 scope.go:122] "RemoveContainer" containerID="158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e" Oct 31 21:55:05.306788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.306751 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:05.306887 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.306844 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:05.306961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.306890 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:05.329244 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:55:05.327691 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7540-79de5a\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:05.423529 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:55:05.423483 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7540-79de5a\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:05.728978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.728891 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx"] Oct 31 21:55:05.729813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.729787 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" containerName="test-curl-7540-79de5a" Oct 31 21:55:05.729813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.729812 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" containerName="test-curl-7540-79de5a" Oct 31 21:55:05.730035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.730021 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" containerName="test-curl-7540-79de5a" Oct 31 21:55:05.733107 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.733081 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.746917 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:55:05.746848 4294 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7540-79de5a\" is forbidden: User \"system:node:ip-10-0-136-156.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-156.ec2.internal' and this object" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:05.748356 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.748332 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-56a6d4\"" Oct 31 21:55:05.748470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.748419 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"routing-overlay\"" Oct 31 21:55:05.748540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.748510 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"openshift-service-ca.crt\"" Oct 31 21:55:05.748629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.748591 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"kube-root-ca.crt\"" Oct 31 21:55:05.751971 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.751735 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx"] Oct 31 21:55:05.817504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.817469 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"agc-cred-8162ebadb51a\"" Oct 31 21:55:05.817688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.817556 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-e2e--34b55c0e-dockercfg-5bdlf\"" Oct 31 21:55:05.840500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.840465 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.840664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.840537 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkt5\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-kube-api-access-mbkt5\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.840836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.840809 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.840892 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.840857 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.948618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.948565 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.949516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.949486 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkt5\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-kube-api-access-mbkt5\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.949934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.949912 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.950989 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.950970 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.967379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.950904 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.967379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.951799 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.967379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.955971 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:05.988767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:05.988733 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkt5\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-kube-api-access-mbkt5\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:06.052777 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.052746 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:06.184097 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.184064 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2"] Oct 31 21:55:06.188066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.188041 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.209012 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.206726 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-fac043-openshift-default-dockercfg-4n97g\"" Oct 31 21:55:06.212335 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.212316 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2"] Oct 31 21:55:06.259258 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.259164 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-envoy\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.259562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.259536 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-credential-socket\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.259922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.259902 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-token\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.260100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.260082 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-certs\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.260296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.260279 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-socket\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.260476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.260457 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-data\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.260638 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.260619 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nt8l\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-kube-api-access-2nt8l\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.260861 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.260840 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-podinfo\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.261112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.261090 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fb4781-a981-49f3-ae57-b638a9dff4bb-istiod-ca-cert\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365225 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-envoy\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365291 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-credential-socket\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365482 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-token\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365508 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-certs\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365572 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-socket\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365598 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-data\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365640 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nt8l\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-kube-api-access-2nt8l\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365715 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-podinfo\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.367333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.365800 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fb4781-a981-49f3-ae57-b638a9dff4bb-istiod-ca-cert\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.368225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.367804 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fb4781-a981-49f3-ae57-b638a9dff4bb-istiod-ca-cert\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.380408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.373219 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-socket\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.380408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.373466 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-credential-socket\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.380408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.373953 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-certs\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.380408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.374390 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-data\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.383031 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.382940 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx"] Oct 31 21:55:06.442324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.420019 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" event={"ID":"fc39eac6-d68f-4420-87d7-48f4c0fbbe96","Type":"ContainerStarted","Data":"d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331"} Oct 31 21:55:06.442324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.421843 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-token\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.442324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.439887 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-podinfo\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.492940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.449750 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-envoy\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.492940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.476015 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nt8l\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-kube-api-access-2nt8l\") pod \"e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.492940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.476367 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:55:06.497341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.496512 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 31 21:55:06.497341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.496571 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 31 21:55:06.527981 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.526939 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:06.965766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.965736 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2"] Oct 31 21:55:06.966982 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:55:06.966959 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fb4781_a981_49f3_ae57_b638a9dff4bb.slice/crio-2ef3e901bd56a7fd76f59d946b41eeac216e396c388642c499b568e2d21514a1 WatchSource:0}: Error finding container 2ef3e901bd56a7fd76f59d946b41eeac216e396c388642c499b568e2d21514a1: Status 404 returned error can't find the container with id 2ef3e901bd56a7fd76f59d946b41eeac216e396c388642c499b568e2d21514a1 Oct 31 21:55:06.970718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.970681 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:06.970801 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.970765 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:06.970842 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:06.970803 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:07.435366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.434229 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" event={"ID":"82fb4781-a981-49f3-ae57-b638a9dff4bb","Type":"ContainerStarted","Data":"c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a"} Oct 31 21:55:07.435366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.434293 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" event={"ID":"82fb4781-a981-49f3-ae57-b638a9dff4bb","Type":"ContainerStarted","Data":"2ef3e901bd56a7fd76f59d946b41eeac216e396c388642c499b568e2d21514a1"} Oct 31 21:55:07.438027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.437904 4294 generic.go:356] "Generic (PLEG): container finished" podID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" containerID="cdbf0b445b3f1b034777b0173039528b3b7dcff6a81fe953bd4399f8bf9e8626" exitCode=137 Oct 31 21:55:07.441289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.441092 4294 generic.go:356] "Generic (PLEG): container finished" podID="66746da2-a008-48d8-a453-8349c639889c" containerID="3bc173ed94383494ba57a55aca4072e7aaf23317b7f548afb58b921aecd1b916" exitCode=0 Oct 31 21:55:07.441289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.441221 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" event={"ID":"66746da2-a008-48d8-a453-8349c639889c","Type":"ContainerDied","Data":"3bc173ed94383494ba57a55aca4072e7aaf23317b7f548afb58b921aecd1b916"} Oct 31 21:55:07.446270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.446148 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" event={"ID":"f4509a71-eb4a-4251-a96a-f7d406fdf029","Type":"ContainerStarted","Data":"6862676aa2c13b18165b75b275aba4fabc1ace673a4b095b74ecb13cd8f239ba"} Oct 31 21:55:07.446270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.446196 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" event={"ID":"f4509a71-eb4a-4251-a96a-f7d406fdf029","Type":"ContainerStarted","Data":"80df0af38956123ff1ef47cd6560a9a77fcf200f0487a6a4beda56c61e4e21d5"} Oct 31 21:55:07.446270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.446223 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:07.519715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.519678 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 31 21:55:07.519870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.519739 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 31 21:55:07.528317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.528275 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:07.563390 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.563342 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podStartSLOduration=1.5633261429999998 podStartE2EDuration="1.563326143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:07.518748868 +0000 UTC m=+2347.643856100" watchObservedRunningTime="2026-10-31 21:55:07.563326143 +0000 UTC m=+2347.688433374" Oct 31 21:55:07.563779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.563746 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" podStartSLOduration=2.563735162 podStartE2EDuration="2.563735162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:07.561509837 +0000 UTC m=+2347.686617069" watchObservedRunningTime="2026-10-31 21:55:07.563735162 +0000 UTC m=+2347.688842409" Oct 31 21:55:07.639607 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.639566 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:55:07.645066 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.645041 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:07.735398 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.735354 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-certs\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.736075 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.736043 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66746da2-a008-48d8-a453-8349c639889c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66746da2-a008-48d8-a453-8349c639889c-istio-podinfo\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.736350 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.736328 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-socket\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.736567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.736533 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-credential-socket\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.736796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.736777 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5-kube-api-access-9zqqp\" (UniqueName: \"kubernetes.io/projected/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5-kube-api-access-9zqqp\") pod \"c82d32b8-6939-4d1f-b5d8-269e4cfcebe5\" (UID: \"c82d32b8-6939-4d1f-b5d8-269e4cfcebe5\") " Oct 31 21:55:07.737041 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.737017 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-envoy\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.737186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.737170 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-istio-token\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-istio-token\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.737402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.737384 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-data\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.743329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.737528 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66746da2-a008-48d8-a453-8349c639889c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66746da2-a008-48d8-a453-8349c639889c-istiod-ca-cert\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.743625 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.743601 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-kube-api-access-2ck6z\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-kube-api-access-2ck6z\") pod \"66746da2-a008-48d8-a453-8349c639889c\" (UID: \"66746da2-a008-48d8-a453-8349c639889c\") " Oct 31 21:55:07.758401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.735943 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-certs" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:07.758401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.751393 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-data" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:07.758401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.751910 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-socket" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:07.758401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.752236 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66746da2-a008-48d8-a453-8349c639889c-istiod-ca-cert" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:55:07.758401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.758374 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-credential-socket" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:07.764504 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.764428 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-istio-token" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:55:07.764855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.764830 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-kube-api-access-2ck6z" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "kube-api-access-2ck6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:55:07.765450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.765424 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5-kube-api-access-9zqqp" pod "c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" (UID: "c82d32b8-6939-4d1f-b5d8-269e4cfcebe5"). InnerVolumeSpecName "kube-api-access-9zqqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:55:07.765955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.765933 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66746da2-a008-48d8-a453-8349c639889c-istio-podinfo" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:55:07.767457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.767414 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-envoy" pod "66746da2-a008-48d8-a453-8349c639889c" (UID: "66746da2-a008-48d8-a453-8349c639889c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:55:07.845811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845781 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.845811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845807 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.845811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845818 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zqqp\" (UniqueName: \"kubernetes.io/projected/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5-kube-api-access-9zqqp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.846055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845827 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.846055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845835 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.846055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845843 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.846055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845851 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66746da2-a008-48d8-a453-8349c639889c-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.846055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845859 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ck6z\" (UniqueName: \"kubernetes.io/projected/66746da2-a008-48d8-a453-8349c639889c-kube-api-access-2ck6z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.846055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845868 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66746da2-a008-48d8-a453-8349c639889c-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:07.846055 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.845875 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66746da2-a008-48d8-a453-8349c639889c-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:55:08.000337 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:07.997720 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82d32b8-6939-4d1f-b5d8-269e4cfcebe5" path="/var/lib/kubelet/pods/c82d32b8-6939-4d1f-b5d8-269e4cfcebe5/volumes" Oct 31 21:55:08.452186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.452102 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" Oct 31 21:55:08.452186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.452120 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx" event={"ID":"66746da2-a008-48d8-a453-8349c639889c","Type":"ContainerDied","Data":"eae5e67fbfc7ae2d309051c979cd235ff427d113be35262a62a099f87ae0fc80"} Oct 31 21:55:08.452186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.452183 4294 scope.go:122] "RemoveContainer" containerID="3bc173ed94383494ba57a55aca4072e7aaf23317b7f548afb58b921aecd1b916" Oct 31 21:55:08.454418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.454388 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7540-79de5a" Oct 31 21:55:08.463427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.463403 4294 scope.go:122] "RemoveContainer" containerID="cdbf0b445b3f1b034777b0173039528b3b7dcff6a81fe953bd4399f8bf9e8626" Oct 31 21:55:08.491326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.489017 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 31 21:55:08.491326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.489074 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 31 21:55:08.514436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.514402 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx"] Oct 31 21:55:08.527750 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.527718 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-tvjtx"] Oct 31 21:55:08.533956 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:08.533935 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:09.462760 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:09.462727 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:09.463658 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:09.463642 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:55:09.496364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:09.491395 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 31 21:55:09.496364 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:09.491452 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 31 21:55:10.005023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:10.004987 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66746da2-a008-48d8-a453-8349c639889c" path="/var/lib/kubelet/pods/66746da2-a008-48d8-a453-8349c639889c/volumes" Oct 31 21:55:10.486388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:10.476686 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 31 21:55:10.486388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:10.476744 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 31 21:55:11.490328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.476711 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 31 21:55:11.490328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.476776 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 31 21:55:11.902486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.902403 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8"] Oct 31 21:55:11.903472 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.903450 4294 cpu_manager.go:414] "containerMap: removing container" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" Oct 31 21:55:11.903528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.903476 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" Oct 31 21:55:11.903651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.903640 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="66746da2-a008-48d8-a453-8349c639889c" containerName="istio-proxy" Oct 31 21:55:11.907647 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.907619 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:11.934283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.934247 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8"] Oct 31 21:55:11.945354 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.945288 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-fac043\"" Oct 31 21:55:11.945540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.945507 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-fac043-dockercfg-dgx5t\"" Oct 31 21:55:11.957367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.957323 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv299\" (UniqueName: \"kubernetes.io/projected/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-kube-api-access-bv299\") pod \"payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:11.957728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:11.957704 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:12.067382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.063627 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv299\" (UniqueName: \"kubernetes.io/projected/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-kube-api-access-bv299\") pod \"payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:12.067382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.063712 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:12.067723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.067695 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:12.105531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.105499 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv299\" (UniqueName: \"kubernetes.io/projected/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-kube-api-access-bv299\") pod \"payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:12.121477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.111595 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn"] Oct 31 21:55:12.121477 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.116122 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.128853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.128821 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn"] Oct 31 21:55:12.167217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.166883 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9035b707-e450-489b-b776-f695d7e1475a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.167414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.167278 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54xpb\" (UniqueName: \"kubernetes.io/projected/9035b707-e450-489b-b776-f695d7e1475a-kube-api-access-54xpb\") pod \"payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.234130 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.234094 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:12.274414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.270010 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54xpb\" (UniqueName: \"kubernetes.io/projected/9035b707-e450-489b-b776-f695d7e1475a-kube-api-access-54xpb\") pod \"payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.274414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.270238 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9035b707-e450-489b-b776-f695d7e1475a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.274414 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.271361 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9035b707-e450-489b-b776-f695d7e1475a-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.358340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.353905 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54xpb\" (UniqueName: \"kubernetes.io/projected/9035b707-e450-489b-b776-f695d7e1475a-kube-api-access-54xpb\") pod \"payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.471376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.470717 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:12.581688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.575779 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:55:12.581688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.577776 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:55:12.581688 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:12.578702 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 21:55:13.099422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.081983 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn"] Oct 31 21:55:13.181359 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.170013 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8"] Oct 31 21:55:13.577700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.577603 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" event={"ID":"9035b707-e450-489b-b776-f695d7e1475a","Type":"ContainerStarted","Data":"5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2"} Oct 31 21:55:13.577700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.577668 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" event={"ID":"9035b707-e450-489b-b776-f695d7e1475a","Type":"ContainerStarted","Data":"63d8e34a44c70dc55dfe91bd3f8e7c4715162c7ac594f6edbe7b68a0497785bb"} Oct 31 21:55:13.577700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.577698 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:13.592013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.591974 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" event={"ID":"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4","Type":"ContainerStarted","Data":"c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824"} Oct 31 21:55:13.592495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.592073 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" event={"ID":"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4","Type":"ContainerStarted","Data":"2c470970a1e538d519e4595482cef6820d8a8e1afd7ca0385975fa154370f259"} Oct 31 21:55:13.592495 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.592249 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:13.618223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.618171 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" podStartSLOduration=2.6181588810000003 podStartE2EDuration="2.618158881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:13.617775406 +0000 UTC m=+2353.742882658" watchObservedRunningTime="2026-10-31 21:55:13.618158881 +0000 UTC m=+2353.743266111" Oct 31 21:55:13.669493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:13.668540 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" podStartSLOduration=2.668521602 podStartE2EDuration="2.668521602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:13.66465374 +0000 UTC m=+2353.789760972" watchObservedRunningTime="2026-10-31 21:55:13.668521602 +0000 UTC m=+2353.793628836" Oct 31 21:55:18.457841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:18.457813 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:55:22.701616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:22.701583 4294 generic.go:356] "Generic (PLEG): container finished" podID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerID="9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41" exitCode=0 Oct 31 21:55:22.702071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:22.701623 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" event={"ID":"45584a2a-6fd4-45fe-8b86-6add93774c95","Type":"ContainerDied","Data":"9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41"} Oct 31 21:55:22.702071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:22.702038 4294 scope.go:122] "RemoveContainer" containerID="9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41" Oct 31 21:55:22.703684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:22.703659 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:22.703751 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:22.703729 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:22.703789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:22.703762 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:23.710865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:23.710828 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" event={"ID":"45584a2a-6fd4-45fe-8b86-6add93774c95","Type":"ContainerStarted","Data":"eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52"} Oct 31 21:55:24.626938 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:24.626908 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 21:55:24.627129 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:24.627073 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 21:55:24.663764 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:24.663730 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:55:24.664921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:24.664885 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 21:55:24.665058 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:24.664950 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 21:55:25.533192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.533148 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j"] Oct 31 21:55:25.539788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.539763 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.563206 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.563180 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j"] Oct 31 21:55:25.570275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.570249 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4a73b1-openshift-default-dockercfg-b2hlf\"" Oct 31 21:55:25.641718 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.641687 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955053d6-46c4-4bc9-b339-c79676dc349f-istiod-ca-cert\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.641898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.641729 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-data\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.641898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.641755 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-socket\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.641898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.641823 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-istio-token\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.642035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.641925 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-envoy\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.642035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.641965 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955053d6-46c4-4bc9-b339-c79676dc349f-istio-podinfo\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.642112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.642079 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-certs\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.642174 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.642157 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr4pd\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-kube-api-access-qr4pd\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.642390 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.642374 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-credential-socket\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.663855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.663802 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 21:55:25.664020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.663993 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 21:55:25.759326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.749121 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr4pd\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-kube-api-access-qr4pd\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.759811 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.759776 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-credential-socket\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.760100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.760077 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955053d6-46c4-4bc9-b339-c79676dc349f-istiod-ca-cert\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.760255 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.760240 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-data\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.775323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.763966 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-credential-socket\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.775323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.764919 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955053d6-46c4-4bc9-b339-c79676dc349f-istiod-ca-cert\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.775323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.765153 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-data\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.775323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.769648 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-socket\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.775697 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.775668 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-socket\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.775831 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.775814 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-istio-token\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.775937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.775924 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-envoy\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.776097 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.776081 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955053d6-46c4-4bc9-b339-c79676dc349f-istio-podinfo\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.776218 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.776205 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-certs\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.776730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.776708 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-certs\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.789701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.789627 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-envoy\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.793329 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.792260 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr4pd\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-kube-api-access-qr4pd\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.816333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.816002 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955053d6-46c4-4bc9-b339-c79676dc349f-istio-podinfo\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.816333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.816227 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-istio-token\") pod \"e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:25.862929 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:25.862630 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:26.292366 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.291348 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j"] Oct 31 21:55:26.298330 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:55:26.295456 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955053d6_46c4_4bc9_b339_c79676dc349f.slice/crio-c500f1a0ee7ae95403068e8c696aaae833e8bdaf355e2ab465ee029af2a30f36 WatchSource:0}: Error finding container c500f1a0ee7ae95403068e8c696aaae833e8bdaf355e2ab465ee029af2a30f36: Status 404 returned error can't find the container with id c500f1a0ee7ae95403068e8c696aaae833e8bdaf355e2ab465ee029af2a30f36 Oct 31 21:55:26.303452 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.303418 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:26.303538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.303488 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:26.303538 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.303518 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:26.664646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.664552 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 21:55:26.664646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.664617 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 21:55:26.675325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.674530 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:55:26.756457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.756413 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" event={"ID":"955053d6-46c4-4bc9-b339-c79676dc349f","Type":"ContainerStarted","Data":"0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a"} Oct 31 21:55:26.756634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.756485 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" event={"ID":"955053d6-46c4-4bc9-b339-c79676dc349f","Type":"ContainerStarted","Data":"c500f1a0ee7ae95403068e8c696aaae833e8bdaf355e2ab465ee029af2a30f36"} Oct 31 21:55:26.863867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:26.863821 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:27.664051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:27.664014 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 21:55:27.664231 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:27.664074 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 21:55:27.870419 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:27.870379 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:27.943496 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:27.943398 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" podStartSLOduration=2.9433836700000002 podStartE2EDuration="2.94338367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:55:26.823399416 +0000 UTC m=+2366.948506648" watchObservedRunningTime="2026-10-31 21:55:27.94338367 +0000 UTC m=+2368.068490974" Oct 31 21:55:28.663674 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:28.663634 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 31 21:55:28.663906 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:28.663719 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 31 21:55:28.770341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:28.770296 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:28.775326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:28.771588 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:55:29.741226 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:29.741180 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:55:29.743254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:29.743227 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 21:55:31.479788 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:55:31.479748 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:55:55.979700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:55.979667 4294 generic.go:356] "Generic (PLEG): container finished" podID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerID="e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da" exitCode=0 Oct 31 21:55:55.994509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:55.994471 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" event={"ID":"4611dd89-b726-40e0-a371-9ca273fb0b8c","Type":"ContainerDied","Data":"e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da"} Oct 31 21:55:55.994879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:55.994861 4294 scope.go:122] "RemoveContainer" containerID="e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da" Oct 31 21:55:55.996622 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:55.996595 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:55.996689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:55.996658 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:55.996728 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:55.996687 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:55:57.002680 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:57.002645 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" event={"ID":"4611dd89-b726-40e0-a371-9ca273fb0b8c","Type":"ContainerStarted","Data":"7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608"} Oct 31 21:55:57.096540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:57.096486 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:55:57.096809 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:57.096778 4294 patch_prober.go:29] interesting pod/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 21:55:57.096918 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:57.096836 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 21:55:58.096735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:58.096695 4294 patch_prober.go:29] interesting pod/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 21:55:58.097268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:58.096763 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 21:55:59.096730 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:59.096681 4294 patch_prober.go:29] interesting pod/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 21:55:59.096941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:55:59.096754 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 21:56:00.096816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:00.096725 4294 patch_prober.go:29] interesting pod/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 21:56:00.096816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:00.096797 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 21:56:00.336640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:00.336610 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:56:00.343204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:00.343182 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 21:56:01.096836 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:01.096797 4294 patch_prober.go:29] interesting pod/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 21:56:01.097024 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:01.096875 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 21:56:01.416647 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:56:01.416561 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:56:01.886509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:01.886477 4294 scope.go:122] "RemoveContainer" containerID="eb9199a404bfc365c3913e60e773980b63827aa5292d176e7a85a7911a0a98b5" Oct 31 21:56:02.284757 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:02.284728 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:56:02.285201 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:02.285181 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:56:02.286151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:02.286128 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:56:07.991363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:07.991328 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz"] Oct 31 21:56:07.996444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:07.996418 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.012326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.012276 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz"] Oct 31 21:56:08.020534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.020511 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f1128d\"/\"autores-model-f1128d-kserve-self-signed-certs\"" Oct 31 21:56:08.020689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.020625 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f1128d\"/\"openshift-service-ca.crt\"" Oct 31 21:56:08.023983 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.023963 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-f1128d\"/\"kube-root-ca.crt\"" Oct 31 21:56:08.023983 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.023965 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-f1128d\"/\"default-dockercfg-dbmcw\"" Oct 31 21:56:08.121491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.121455 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-model-cache\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.121700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.121600 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crjjl\" (UniqueName: \"kubernetes.io/projected/e18a1502-aa87-4610-a89d-700f89b5b25a-kube-api-access-crjjl\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.121700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.121648 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18a1502-aa87-4610-a89d-700f89b5b25a-tls-certs\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.121700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.121669 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-dshm\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.121884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.121747 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-tmp-dir\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.121884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.121786 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-home\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.223797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.223765 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-model-cache\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.223974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.223867 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crjjl\" (UniqueName: \"kubernetes.io/projected/e18a1502-aa87-4610-a89d-700f89b5b25a-kube-api-access-crjjl\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.223974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.223894 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18a1502-aa87-4610-a89d-700f89b5b25a-tls-certs\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.223974 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.223911 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-dshm\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.224216 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.224171 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-tmp-dir\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.224388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.224249 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-home\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.224388 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.224247 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-model-cache\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.224535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.224512 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-tmp-dir\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.224675 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.224647 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-home\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.226386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.226358 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-dshm\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.226595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.226576 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18a1502-aa87-4610-a89d-700f89b5b25a-tls-certs\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.260371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.260051 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crjjl\" (UniqueName: \"kubernetes.io/projected/e18a1502-aa87-4610-a89d-700f89b5b25a-kube-api-access-crjjl\") pod \"autores-model-f1128d-kserve-7b65f4f845-2ntzz\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.319324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.315404 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:08.674853 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:08.674824 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz"] Oct 31 21:56:08.675866 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:56:08.675841 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode18a1502_aa87_4610_a89d_700f89b5b25a.slice/crio-9bea7c62f2d4b830ceaf638cb8e707ee566bca0ad75458ed803c7f296fdf1c83 WatchSource:0}: Error finding container 9bea7c62f2d4b830ceaf638cb8e707ee566bca0ad75458ed803c7f296fdf1c83: Status 404 returned error can't find the container with id 9bea7c62f2d4b830ceaf638cb8e707ee566bca0ad75458ed803c7f296fdf1c83 Oct 31 21:56:09.094454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:09.094417 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" event={"ID":"e18a1502-aa87-4610-a89d-700f89b5b25a","Type":"ContainerStarted","Data":"34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45"} Oct 31 21:56:09.094454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:09.094460 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" event={"ID":"e18a1502-aa87-4610-a89d-700f89b5b25a","Type":"ContainerStarted","Data":"9bea7c62f2d4b830ceaf638cb8e707ee566bca0ad75458ed803c7f296fdf1c83"} Oct 31 21:56:09.136120 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:09.136063 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" podStartSLOduration=2.136049573 podStartE2EDuration="2.136049573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:56:09.134804587 +0000 UTC m=+2409.259911818" watchObservedRunningTime="2026-10-31 21:56:09.136049573 +0000 UTC m=+2409.261156804" Oct 31 21:56:09.316197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:09.316167 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:18.316422 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.316383 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:18.331580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.331552 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:18.576963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.576881 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk"] Oct 31 21:56:18.582840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.582817 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.600710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.599638 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"payload-processing-external-model-e2e--bb6f7797-dockercfg-6p5zs\"" Oct 31 21:56:18.600710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.600327 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"agc-cred-0185341c205f\"" Oct 31 21:56:18.600710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.600393 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c34cbd\"" Oct 31 21:56:18.600710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.600444 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"agc-cred-3d39da95e8af\"" Oct 31 21:56:18.600710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.600524 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"routing-overlay\"" Oct 31 21:56:18.600710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.600580 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"openshift-service-ca.crt\"" Oct 31 21:56:18.600710 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.600664 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"kube-root-ca.crt\"" Oct 31 21:56:18.603475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.603446 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk"] Oct 31 21:56:18.646621 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.646585 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.646822 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.646738 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.646896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.646825 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc9tb\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-kube-api-access-jc9tb\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.646896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.646882 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.748213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.748163 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.748467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.748263 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.748467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.748333 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc9tb\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-kube-api-access-jc9tb\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.748467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.748370 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.749145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.749050 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.749275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.749170 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.751368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.751342 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.772676 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.772644 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc9tb\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-kube-api-access-jc9tb\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:18.906463 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:18.906377 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:19.220185 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:19.220159 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk"] Oct 31 21:56:19.221287 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:56:19.221263 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd2e103_bd93_4960_b392_5c0e4d292da6.slice/crio-9e94359109ccc043927f45c121b17d576d159b949c18ca21b817c4f7ce212651 WatchSource:0}: Error finding container 9e94359109ccc043927f45c121b17d576d159b949c18ca21b817c4f7ce212651: Status 404 returned error can't find the container with id 9e94359109ccc043927f45c121b17d576d159b949c18ca21b817c4f7ce212651 Oct 31 21:56:19.229382 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:19.229362 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:20.227195 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:20.227160 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" event={"ID":"dbd2e103-bd93-4960-b392-5c0e4d292da6","Type":"ContainerStarted","Data":"af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801"} Oct 31 21:56:20.227715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:20.227220 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" event={"ID":"dbd2e103-bd93-4960-b392-5c0e4d292da6","Type":"ContainerStarted","Data":"9e94359109ccc043927f45c121b17d576d159b949c18ca21b817c4f7ce212651"} Oct 31 21:56:20.227715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:20.227462 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:20.273865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:20.273780 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" podStartSLOduration=2.273767552 podStartE2EDuration="2.273767552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:56:20.273524642 +0000 UTC m=+2420.398631875" watchObservedRunningTime="2026-10-31 21:56:20.273767552 +0000 UTC m=+2420.398874782" Oct 31 21:56:21.033404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.033376 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8"] Oct 31 21:56:21.098509 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.044689 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3134895-nn4c8"] Oct 31 21:56:21.235043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.235018 4294 generic.go:356] "Generic (PLEG): container finished" podID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerID="c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a" exitCode=0 Oct 31 21:56:21.235447 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.235087 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" event={"ID":"82fb4781-a981-49f3-ae57-b638a9dff4bb","Type":"ContainerDied","Data":"c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a"} Oct 31 21:56:21.235566 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.235549 4294 scope.go:122] "RemoveContainer" containerID="c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a" Oct 31 21:56:21.237227 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.237200 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:21.237322 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.237272 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:21.237369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:21.237352 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:22.002409 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:22.002370 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892bf842-69b8-49d1-b64b-adb78810ba7f" path="/var/lib/kubelet/pods/892bf842-69b8-49d1-b64b-adb78810ba7f/volumes" Oct 31 21:56:22.243719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:22.243680 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" event={"ID":"82fb4781-a981-49f3-ae57-b638a9dff4bb","Type":"ContainerStarted","Data":"986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3"} Oct 31 21:56:22.527903 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:22.527868 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:56:22.528985 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:22.528957 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 21:56:22.529111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:22.529010 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 21:56:23.528636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:23.528602 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 21:56:23.529018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:23.528655 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 21:56:24.463479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:24.463447 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:56:24.537620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:24.537589 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 21:56:24.538243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:24.538210 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 21:56:25.528615 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:25.528581 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 21:56:25.528796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:25.528631 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 21:56:26.528816 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:26.528776 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 31 21:56:26.529225 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:26.528832 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 31 21:56:27.533340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:27.533288 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:56:27.534259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:27.534236 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 21:56:28.224712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.224677 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt"] Oct 31 21:56:28.229552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.229525 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.250643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.250614 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt"] Oct 31 21:56:28.257940 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.257908 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2dbae9-openshift-default-dockercfg-9kfk8\"" Oct 31 21:56:28.316975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.316935 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istiod-ca-cert\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317059 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-data\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317182 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-podinfo\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317290 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317231 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-certs\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317290 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317260 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-envoy\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317416 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5w88\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-kube-api-access-b5w88\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317534 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-socket\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317621 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317610 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-credential-socket\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.317687 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.317670 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-token\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.420030 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-token\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.432689 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istiod-ca-cert\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.432767 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-data\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.432925 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-podinfo\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.432962 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-certs\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.432995 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-envoy\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.433064 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5w88\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-kube-api-access-b5w88\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.433260 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-socket\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.433355 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-credential-socket\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.434951 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.434349 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-socket\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.452324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.435515 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istiod-ca-cert\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.452324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.435750 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-data\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.452324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.444052 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-credential-socket\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.452324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.444277 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-certs\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.472326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.471548 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5w88\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-kube-api-access-b5w88\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.481194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.480968 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-envoy\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.503324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.501903 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-token\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.515729 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.515696 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-podinfo\") pod \"e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.551917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.551657 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:28.924220 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.924195 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt"] Oct 31 21:56:28.926060 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:56:28.926027 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b82ddc_e4ee_40fc_ab6b_e45b30ba9573.slice/crio-1aca8571888123472b29515a80ade6c9f7cdaf82041b5116c45b171e7b00e24e WatchSource:0}: Error finding container 1aca8571888123472b29515a80ade6c9f7cdaf82041b5116c45b171e7b00e24e: Status 404 returned error can't find the container with id 1aca8571888123472b29515a80ade6c9f7cdaf82041b5116c45b171e7b00e24e Oct 31 21:56:28.928825 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.928793 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:28.928907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.928861 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:28.928907 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:28.928890 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:29.298555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:29.298516 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" event={"ID":"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573","Type":"ContainerStarted","Data":"4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b"} Oct 31 21:56:29.298759 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:29.298588 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" event={"ID":"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573","Type":"ContainerStarted","Data":"1aca8571888123472b29515a80ade6c9f7cdaf82041b5116c45b171e7b00e24e"} Oct 31 21:56:29.357578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:29.357522 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podStartSLOduration=1.357506721 podStartE2EDuration="1.357506721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:56:29.354466486 +0000 UTC m=+2429.479573718" watchObservedRunningTime="2026-10-31 21:56:29.357506721 +0000 UTC m=+2429.482613951" Oct 31 21:56:29.553194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:29.553105 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:29.559210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:29.559186 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:30.305251 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:30.305221 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:30.306077 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:30.306058 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:56:31.238408 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:31.238373 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:56:31.320935 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:56:31.320902 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:56:32.041700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:32.041661 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t"] Oct 31 21:56:32.046442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:32.046416 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd8520a8-wzq6t"] Oct 31 21:56:32.188178 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:32.188136 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx"] Oct 31 21:56:33.986702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:33.986668 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fbc5f98-973f-445a-ad11-84ca7703443d" path="/var/lib/kubelet/pods/7fbc5f98-973f-445a-ad11-84ca7703443d/volumes" Oct 31 21:56:34.241606 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.241518 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz"] Oct 31 21:56:34.508468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.508391 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82"] Oct 31 21:56:34.513029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.513011 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.533468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.533438 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82"] Oct 31 21:56:34.548840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.548818 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:56:34.548840 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.548818 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:56:34.645453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.645418 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009e5e96-81df-48f0-8701-4125535b8566-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.645665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.645650 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009e5e96-81df-48f0-8701-4125535b8566-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.645762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.645744 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft4sr\" (UniqueName: \"kubernetes.io/projected/009e5e96-81df-48f0-8701-4125535b8566-kube-api-access-ft4sr\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.747357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.747282 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009e5e96-81df-48f0-8701-4125535b8566-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.747532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.747513 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009e5e96-81df-48f0-8701-4125535b8566-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.747578 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.747572 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft4sr\" (UniqueName: \"kubernetes.io/projected/009e5e96-81df-48f0-8701-4125535b8566-kube-api-access-ft4sr\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.748042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.748010 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009e5e96-81df-48f0-8701-4125535b8566-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.749920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.749900 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009e5e96-81df-48f0-8701-4125535b8566-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.771039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.770971 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft4sr\" (UniqueName: \"kubernetes.io/projected/009e5e96-81df-48f0-8701-4125535b8566-kube-api-access-ft4sr\") pod \"maas-api-revoke-keys-e2e-autores-f1128d-dkt82\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:34.828345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:34.828285 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:35.133435 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:35.133412 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82"] Oct 31 21:56:35.134117 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:56:35.134092 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009e5e96_81df_48f0_8701_4125535b8566.slice/crio-ca26ceb358579fa6daf13c1736bb5f2dad0abab4c73ac241a7752eea5db14266 WatchSource:0}: Error finding container ca26ceb358579fa6daf13c1736bb5f2dad0abab4c73ac241a7752eea5db14266: Status 404 returned error can't find the container with id ca26ceb358579fa6daf13c1736bb5f2dad0abab4c73ac241a7752eea5db14266 Oct 31 21:56:35.349673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:35.349640 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" event={"ID":"009e5e96-81df-48f0-8701-4125535b8566","Type":"ContainerStarted","Data":"ca26ceb358579fa6daf13c1736bb5f2dad0abab4c73ac241a7752eea5db14266"} Oct 31 21:56:36.358143 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:36.358062 4294 generic.go:356] "Generic (PLEG): container finished" podID="009e5e96-81df-48f0-8701-4125535b8566" containerID="d34d6190f336ffebe6de7e36e2a8655f381e2e996ff4f795c032b6ea113b788d" exitCode=0 Oct 31 21:56:36.358523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:36.358145 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" event={"ID":"009e5e96-81df-48f0-8701-4125535b8566","Type":"ContainerDied","Data":"d34d6190f336ffebe6de7e36e2a8655f381e2e996ff4f795c032b6ea113b788d"} Oct 31 21:56:37.188932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.188897 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" podUID="f4509a71-eb4a-4251-a96a-f7d406fdf029" containerName="payload-processing" containerID="cri-o://6862676aa2c13b18165b75b275aba4fabc1ace673a4b095b74ecb13cd8f239ba" gracePeriod=25 Oct 31 21:56:37.367281 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.367251 4294 generic.go:356] "Generic (PLEG): container finished" podID="f4509a71-eb4a-4251-a96a-f7d406fdf029" containerID="6862676aa2c13b18165b75b275aba4fabc1ace673a4b095b74ecb13cd8f239ba" exitCode=0 Oct 31 21:56:37.367805 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.367351 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" event={"ID":"f4509a71-eb4a-4251-a96a-f7d406fdf029","Type":"ContainerDied","Data":"6862676aa2c13b18165b75b275aba4fabc1ace673a4b095b74ecb13cd8f239ba"} Oct 31 21:56:37.471975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.471950 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:56:37.546844 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.546812 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:37.594642 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.594602 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-plugins-config-volume\") pod \"f4509a71-eb4a-4251-a96a-f7d406fdf029\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " Oct 31 21:56:37.595292 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.595166 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-plugins-config-volume" pod "f4509a71-eb4a-4251-a96a-f7d406fdf029" (UID: "f4509a71-eb4a-4251-a96a-f7d406fdf029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:56:37.595457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.595437 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-routing-overlay-volume\") pod \"f4509a71-eb4a-4251-a96a-f7d406fdf029\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " Oct 31 21:56:37.595549 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.595536 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-kube-api-access-mbkt5\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-kube-api-access-mbkt5\") pod \"f4509a71-eb4a-4251-a96a-f7d406fdf029\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " Oct 31 21:56:37.595611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.595574 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-provider-credentials\") pod \"f4509a71-eb4a-4251-a96a-f7d406fdf029\" (UID: \"f4509a71-eb4a-4251-a96a-f7d406fdf029\") " Oct 31 21:56:37.596158 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.596125 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-routing-overlay-volume" pod "f4509a71-eb4a-4251-a96a-f7d406fdf029" (UID: "f4509a71-eb4a-4251-a96a-f7d406fdf029"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:56:37.596473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.596449 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:37.596571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.596477 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4509a71-eb4a-4251-a96a-f7d406fdf029-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:37.598387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.598346 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-provider-credentials" pod "f4509a71-eb4a-4251-a96a-f7d406fdf029" (UID: "f4509a71-eb4a-4251-a96a-f7d406fdf029"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:37.598827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.598801 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-kube-api-access-mbkt5" pod "f4509a71-eb4a-4251-a96a-f7d406fdf029" (UID: "f4509a71-eb4a-4251-a96a-f7d406fdf029"). InnerVolumeSpecName "kube-api-access-mbkt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:37.697791 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.697708 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009e5e96-81df-48f0-8701-4125535b8566-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009e5e96-81df-48f0-8701-4125535b8566-maas-api-openshift-ca\") pod \"009e5e96-81df-48f0-8701-4125535b8566\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " Oct 31 21:56:37.697962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.697798 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009e5e96-81df-48f0-8701-4125535b8566-kube-api-access-ft4sr\" (UniqueName: \"kubernetes.io/projected/009e5e96-81df-48f0-8701-4125535b8566-kube-api-access-ft4sr\") pod \"009e5e96-81df-48f0-8701-4125535b8566\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " Oct 31 21:56:37.697962 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.697868 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/009e5e96-81df-48f0-8701-4125535b8566-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009e5e96-81df-48f0-8701-4125535b8566-maas-api-xks-ca\") pod \"009e5e96-81df-48f0-8701-4125535b8566\" (UID: \"009e5e96-81df-48f0-8701-4125535b8566\") " Oct 31 21:56:37.698138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.698115 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009e5e96-81df-48f0-8701-4125535b8566-maas-api-openshift-ca" pod "009e5e96-81df-48f0-8701-4125535b8566" (UID: "009e5e96-81df-48f0-8701-4125535b8566"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:56:37.698459 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.698443 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009e5e96-81df-48f0-8701-4125535b8566-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:37.698521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.698464 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbkt5\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-kube-api-access-mbkt5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:37.698521 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.698478 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4509a71-eb4a-4251-a96a-f7d406fdf029-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:37.699942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.699920 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009e5e96-81df-48f0-8701-4125535b8566-maas-api-xks-ca" pod "009e5e96-81df-48f0-8701-4125535b8566" (UID: "009e5e96-81df-48f0-8701-4125535b8566"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:56:37.700013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.699985 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009e5e96-81df-48f0-8701-4125535b8566-kube-api-access-ft4sr" pod "009e5e96-81df-48f0-8701-4125535b8566" (UID: "009e5e96-81df-48f0-8701-4125535b8566"). InnerVolumeSpecName "kube-api-access-ft4sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:37.800269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.800230 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft4sr\" (UniqueName: \"kubernetes.io/projected/009e5e96-81df-48f0-8701-4125535b8566-kube-api-access-ft4sr\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:37.800269 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:37.800270 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009e5e96-81df-48f0-8701-4125535b8566-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:38.375142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.375056 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" Oct 31 21:56:38.375142 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.375079 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx" event={"ID":"f4509a71-eb4a-4251-a96a-f7d406fdf029","Type":"ContainerDied","Data":"80df0af38956123ff1ef47cd6560a9a77fcf200f0487a6a4beda56c61e4e21d5"} Oct 31 21:56:38.375702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.375165 4294 scope.go:122] "RemoveContainer" containerID="6862676aa2c13b18165b75b275aba4fabc1ace673a4b095b74ecb13cd8f239ba" Oct 31 21:56:38.377483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.377434 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" event={"ID":"009e5e96-81df-48f0-8701-4125535b8566","Type":"ContainerDied","Data":"ca26ceb358579fa6daf13c1736bb5f2dad0abab4c73ac241a7752eea5db14266"} Oct 31 21:56:38.377483 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.377484 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca26ceb358579fa6daf13c1736bb5f2dad0abab4c73ac241a7752eea5db14266" Oct 31 21:56:38.377672 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.377515 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82" Oct 31 21:56:38.431348 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.431321 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx"] Oct 31 21:56:38.442406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:38.442383 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97tj9dx"] Oct 31 21:56:39.985315 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:39.985278 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4509a71-eb4a-4251-a96a-f7d406fdf029" path="/var/lib/kubelet/pods/f4509a71-eb4a-4251-a96a-f7d406fdf029/volumes" Oct 31 21:56:44.987326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:44.986878 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl"] Oct 31 21:56:45.005325 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:45.004946 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl"] Oct 31 21:56:49.265843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.265795 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" podUID="e18a1502-aa87-4610-a89d-700f89b5b25a" containerName="main" containerID="cri-o://34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45" gracePeriod=45 Oct 31 21:56:49.546166 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.546135 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:49.558236 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.558201 4294 generic.go:356] "Generic (PLEG): container finished" podID="e18a1502-aa87-4610-a89d-700f89b5b25a" containerID="34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45" exitCode=0 Oct 31 21:56:49.558418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.558270 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" Oct 31 21:56:49.558418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.558288 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" event={"ID":"e18a1502-aa87-4610-a89d-700f89b5b25a","Type":"ContainerDied","Data":"34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45"} Oct 31 21:56:49.558418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.558375 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz" event={"ID":"e18a1502-aa87-4610-a89d-700f89b5b25a","Type":"ContainerDied","Data":"9bea7c62f2d4b830ceaf638cb8e707ee566bca0ad75458ed803c7f296fdf1c83"} Oct 31 21:56:49.558418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.558398 4294 scope.go:122] "RemoveContainer" containerID="34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45" Oct 31 21:56:49.568248 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.568233 4294 scope.go:122] "RemoveContainer" containerID="34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45" Oct 31 21:56:49.568576 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:56:49.568551 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45\": container with ID starting with 34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45 not found: ID does not exist" containerID="34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45" Oct 31 21:56:49.568660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.568586 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45"} err="failed to get container status \"34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45\": rpc error: code = NotFound desc = could not find container \"34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45\": container with ID starting with 34f539e4da3fdd6b760235fc16942c88616302c56fe2b07940a2a9972dc17f45 not found: ID does not exist" Oct 31 21:56:49.754640 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.754605 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-tmp-dir\") pod \"e18a1502-aa87-4610-a89d-700f89b5b25a\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " Oct 31 21:56:49.754841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.754683 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-dshm\") pod \"e18a1502-aa87-4610-a89d-700f89b5b25a\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " Oct 31 21:56:49.754841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.754738 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18a1502-aa87-4610-a89d-700f89b5b25a-kube-api-access-crjjl\" (UniqueName: \"kubernetes.io/projected/e18a1502-aa87-4610-a89d-700f89b5b25a-kube-api-access-crjjl\") pod \"e18a1502-aa87-4610-a89d-700f89b5b25a\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " Oct 31 21:56:49.754841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.754834 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-home\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-home\") pod \"e18a1502-aa87-4610-a89d-700f89b5b25a\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " Oct 31 21:56:49.755018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.754948 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-tmp-dir" pod "e18a1502-aa87-4610-a89d-700f89b5b25a" (UID: "e18a1502-aa87-4610-a89d-700f89b5b25a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:49.755018 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.754977 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-model-cache\") pod \"e18a1502-aa87-4610-a89d-700f89b5b25a\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " Oct 31 21:56:49.755146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.755020 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-home" pod "e18a1502-aa87-4610-a89d-700f89b5b25a" (UID: "e18a1502-aa87-4610-a89d-700f89b5b25a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:49.755146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.755042 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18a1502-aa87-4610-a89d-700f89b5b25a-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18a1502-aa87-4610-a89d-700f89b5b25a-tls-certs\") pod \"e18a1502-aa87-4610-a89d-700f89b5b25a\" (UID: \"e18a1502-aa87-4610-a89d-700f89b5b25a\") " Oct 31 21:56:49.755241 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.755192 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-model-cache" pod "e18a1502-aa87-4610-a89d-700f89b5b25a" (UID: "e18a1502-aa87-4610-a89d-700f89b5b25a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:49.755896 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.755871 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:49.756040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.755900 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:49.756040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.755915 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:49.757037 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.757005 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-dshm" pod "e18a1502-aa87-4610-a89d-700f89b5b25a" (UID: "e18a1502-aa87-4610-a89d-700f89b5b25a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:56:49.757150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.757060 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18a1502-aa87-4610-a89d-700f89b5b25a-tls-certs" pod "e18a1502-aa87-4610-a89d-700f89b5b25a" (UID: "e18a1502-aa87-4610-a89d-700f89b5b25a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:56:49.757150 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.757087 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18a1502-aa87-4610-a89d-700f89b5b25a-kube-api-access-crjjl" pod "e18a1502-aa87-4610-a89d-700f89b5b25a" (UID: "e18a1502-aa87-4610-a89d-700f89b5b25a"). InnerVolumeSpecName "kube-api-access-crjjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:49.857182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.857091 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18a1502-aa87-4610-a89d-700f89b5b25a-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:49.857182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.857125 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crjjl\" (UniqueName: \"kubernetes.io/projected/e18a1502-aa87-4610-a89d-700f89b5b25a-kube-api-access-crjjl\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:49.857182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.857137 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18a1502-aa87-4610-a89d-700f89b5b25a-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:49.905536 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.905502 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz"] Oct 31 21:56:49.916944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.916919 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-f1128d/autores-model-f1128d-kserve-7b65f4f845-2ntzz"] Oct 31 21:56:49.986942 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.986909 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18a1502-aa87-4610-a89d-700f89b5b25a" path="/var/lib/kubelet/pods/e18a1502-aa87-4610-a89d-700f89b5b25a/volumes" Oct 31 21:56:49.987243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:49.987225 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" podUID="e72b79fe-e6df-4e6d-86e3-c9c45119f449" containerName="payload-processing" containerID="cri-o://60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19" gracePeriod=25 Oct 31 21:56:50.005661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.005631 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" podUID="bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" containerName="payload-pre-processing" containerID="cri-o://3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10" gracePeriod=25 Oct 31 21:56:50.335570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.335487 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:56:50.373799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.373732 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:56:50.469162 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.469130 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72b79fe-e6df-4e6d-86e3-c9c45119f449-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72b79fe-e6df-4e6d-86e3-c9c45119f449-plugins-config-volume\") pod \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " Oct 31 21:56:50.469363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.469237 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-kube-api-access-lnz77\" (UniqueName: \"kubernetes.io/projected/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-kube-api-access-lnz77\") pod \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " Oct 31 21:56:50.469363 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.469315 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72b79fe-e6df-4e6d-86e3-c9c45119f449-kube-api-access-mcnbj\" (UniqueName: \"kubernetes.io/projected/e72b79fe-e6df-4e6d-86e3-c9c45119f449-kube-api-access-mcnbj\") pod \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\" (UID: \"e72b79fe-e6df-4e6d-86e3-c9c45119f449\") " Oct 31 21:56:50.469484 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.469418 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-plugins-config-volume\") pod \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\" (UID: \"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5\") " Oct 31 21:56:50.469693 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.469665 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72b79fe-e6df-4e6d-86e3-c9c45119f449-plugins-config-volume" pod "e72b79fe-e6df-4e6d-86e3-c9c45119f449" (UID: "e72b79fe-e6df-4e6d-86e3-c9c45119f449"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:56:50.470119 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.470097 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-plugins-config-volume" pod "bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" (UID: "bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:56:50.470358 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.470339 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e72b79fe-e6df-4e6d-86e3-c9c45119f449-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:50.472013 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.471987 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-kube-api-access-lnz77" pod "bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" (UID: "bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5"). InnerVolumeSpecName "kube-api-access-lnz77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:50.472096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.472017 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72b79fe-e6df-4e6d-86e3-c9c45119f449-kube-api-access-mcnbj" pod "e72b79fe-e6df-4e6d-86e3-c9c45119f449" (UID: "e72b79fe-e6df-4e6d-86e3-c9c45119f449"). InnerVolumeSpecName "kube-api-access-mcnbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:56:50.566608 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.566575 4294 generic.go:356] "Generic (PLEG): container finished" podID="e72b79fe-e6df-4e6d-86e3-c9c45119f449" containerID="60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19" exitCode=0 Oct 31 21:56:50.566821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.566624 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" Oct 31 21:56:50.566821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.566655 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" event={"ID":"e72b79fe-e6df-4e6d-86e3-c9c45119f449","Type":"ContainerDied","Data":"60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19"} Oct 31 21:56:50.566821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.566721 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl" event={"ID":"e72b79fe-e6df-4e6d-86e3-c9c45119f449","Type":"ContainerDied","Data":"427490d7abca4850423122f891ae1be3d1f3f0b7cc451c40b7f4f9f5525220d3"} Oct 31 21:56:50.566821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.566755 4294 scope.go:122] "RemoveContainer" containerID="60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19" Oct 31 21:56:50.568567 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.568545 4294 generic.go:356] "Generic (PLEG): container finished" podID="bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" containerID="3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10" exitCode=0 Oct 31 21:56:50.568665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.568614 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" event={"ID":"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5","Type":"ContainerDied","Data":"3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10"} Oct 31 21:56:50.568665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.568656 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" event={"ID":"bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5","Type":"ContainerDied","Data":"58bc079dff831683b7d376106b55164f7cd854f3eb98cb9f4dc246e7baedbdea"} Oct 31 21:56:50.568762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.568722 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl" Oct 31 21:56:50.572783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.572722 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:50.572783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.572747 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnz77\" (UniqueName: \"kubernetes.io/projected/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5-kube-api-access-lnz77\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:50.572783 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.572761 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcnbj\" (UniqueName: \"kubernetes.io/projected/e72b79fe-e6df-4e6d-86e3-c9c45119f449-kube-api-access-mcnbj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:56:50.577881 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.577850 4294 scope.go:122] "RemoveContainer" containerID="60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19" Oct 31 21:56:50.578171 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:56:50.578149 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19\": container with ID starting with 60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19 not found: ID does not exist" containerID="60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19" Oct 31 21:56:50.578236 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.578185 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19"} err="failed to get container status \"60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19\": rpc error: code = NotFound desc = could not find container \"60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19\": container with ID starting with 60b8d8d308d355494720cd4a919ab7a1ecd8d46bad90de09df7eec1a87ee3d19 not found: ID does not exist" Oct 31 21:56:50.578236 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.578204 4294 scope.go:122] "RemoveContainer" containerID="3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10" Oct 31 21:56:50.590341 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.590323 4294 scope.go:122] "RemoveContainer" containerID="3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10" Oct 31 21:56:50.590653 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:56:50.590626 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10\": container with ID starting with 3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10 not found: ID does not exist" containerID="3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10" Oct 31 21:56:50.590745 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.590658 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10"} err="failed to get container status \"3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10\": rpc error: code = NotFound desc = could not find container \"3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10\": container with ID starting with 3563f42c9365b30e74ffc9c0c9b738f8ad83816fc22f0ef6fa9c532a7601fc10 not found: ID does not exist" Oct 31 21:56:50.634952 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.634844 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl"] Oct 31 21:56:50.640198 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.640164 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-f1128d-575cff74f5-j7hbl"] Oct 31 21:56:50.673515 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.673478 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl"] Oct 31 21:56:50.679338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:50.679293 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f1128d-666bb85d4f-nstgl"] Oct 31 21:56:51.987922 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:51.987889 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" path="/var/lib/kubelet/pods/bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5/volumes" Oct 31 21:56:51.988547 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:51.988526 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72b79fe-e6df-4e6d-86e3-c9c45119f449" path="/var/lib/kubelet/pods/e72b79fe-e6df-4e6d-86e3-c9c45119f449/volumes" Oct 31 21:56:53.598418 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:53.598383 4294 generic.go:356] "Generic (PLEG): container finished" podID="955053d6-46c4-4bc9-b339-c79676dc349f" containerID="0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a" exitCode=0 Oct 31 21:56:53.598872 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:53.598439 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" event={"ID":"955053d6-46c4-4bc9-b339-c79676dc349f","Type":"ContainerDied","Data":"0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a"} Oct 31 21:56:53.599016 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:53.598995 4294 scope.go:122] "RemoveContainer" containerID="0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a" Oct 31 21:56:53.601056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:53.601019 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:53.601188 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:53.601092 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:53.601188 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:53.601130 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:56:54.607118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:54.607078 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" event={"ID":"955053d6-46c4-4bc9-b339-c79676dc349f","Type":"ContainerStarted","Data":"a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4"} Oct 31 21:56:54.864254 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:54.864173 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:56:54.865411 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:54.865383 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 21:56:54.865541 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:54.865456 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 21:56:55.209988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.209891 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp"] Oct 31 21:56:55.210516 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210498 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e18a1502-aa87-4610-a89d-700f89b5b25a" containerName="main" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210516 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18a1502-aa87-4610-a89d-700f89b5b25a" containerName="main" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210530 4294 cpu_manager.go:414] "containerMap: removing container" podUID="bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" containerName="payload-pre-processing" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210535 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" containerName="payload-pre-processing" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210551 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e72b79fe-e6df-4e6d-86e3-c9c45119f449" containerName="payload-processing" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210556 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72b79fe-e6df-4e6d-86e3-c9c45119f449" containerName="payload-processing" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210570 4294 cpu_manager.go:414] "containerMap: removing container" podUID="009e5e96-81df-48f0-8701-4125535b8566" containerName="revoke-keys" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210575 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009e5e96-81df-48f0-8701-4125535b8566" containerName="revoke-keys" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210585 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f4509a71-eb4a-4251-a96a-f7d406fdf029" containerName="payload-processing" Oct 31 21:56:55.210614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210589 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4509a71-eb4a-4251-a96a-f7d406fdf029" containerName="payload-processing" Oct 31 21:56:55.210894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210681 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18a1502-aa87-4610-a89d-700f89b5b25a" containerName="main" Oct 31 21:56:55.210894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210690 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfdee3a-e1e6-40b3-9ff3-c3d796cdafd5" containerName="payload-pre-processing" Oct 31 21:56:55.210894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210697 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72b79fe-e6df-4e6d-86e3-c9c45119f449" containerName="payload-processing" Oct 31 21:56:55.210894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210705 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="009e5e96-81df-48f0-8701-4125535b8566" containerName="revoke-keys" Oct 31 21:56:55.210894 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.210712 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4509a71-eb4a-4251-a96a-f7d406fdf029" containerName="payload-processing" Oct 31 21:56:55.213161 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.213143 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.232147 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.232121 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"kube-root-ca.crt\"" Oct 31 21:56:55.232260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.232201 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"routing-overlay\"" Oct 31 21:56:55.232316 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.232274 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"openshift-service-ca.crt\"" Oct 31 21:56:55.232404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.232388 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4a73b1\"" Oct 31 21:56:55.233444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.233410 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp"] Oct 31 21:56:55.295326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.283408 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"payload-processing-external-model-e2e--23c252ba-dockercfg-l8pcd\"" Oct 31 21:56:55.295326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.283521 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"agc-cred-eb8c6251292d\"" Oct 31 21:56:55.307604 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.307573 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.307778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.307617 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gc2\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-kube-api-access-q8gc2\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.307778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.307719 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.307916 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.307902 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.431352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.417260 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.431352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.418333 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.431352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.418375 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gc2\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-kube-api-access-q8gc2\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.431352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.418432 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.431352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.419186 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.431352 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.418151 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.455326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.450268 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.478964 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.477196 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gc2\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-kube-api-access-q8gc2\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.532336 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.530617 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:55.864574 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.864542 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 21:56:55.864977 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.864594 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 21:56:55.898401 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:55.898368 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp"] Oct 31 21:56:55.902035 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:56:55.902004 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4259b89b_6819_4f46_a876_451517fbe9aa.slice/crio-e43840f1dc99955f237d49b8b5b211605652a1b74f25af158ed7cc46bd2864b3 WatchSource:0}: Error finding container e43840f1dc99955f237d49b8b5b211605652a1b74f25af158ed7cc46bd2864b3: Status 404 returned error can't find the container with id e43840f1dc99955f237d49b8b5b211605652a1b74f25af158ed7cc46bd2864b3 Oct 31 21:56:56.633165 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:56.633123 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" event={"ID":"4259b89b-6819-4f46-a876-451517fbe9aa","Type":"ContainerStarted","Data":"2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952"} Oct 31 21:56:56.633373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:56.633211 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" event={"ID":"4259b89b-6819-4f46-a876-451517fbe9aa","Type":"ContainerStarted","Data":"e43840f1dc99955f237d49b8b5b211605652a1b74f25af158ed7cc46bd2864b3"} Oct 31 21:56:56.633373 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:56.633240 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:56:56.863960 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:56.863922 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 21:56:56.864137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:56.863980 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 21:56:57.863948 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:57.863914 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 21:56:57.864378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:57.863966 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 21:56:58.771081 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:58.771050 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:56:58.863657 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:58.863623 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body= Oct 31 21:56:58.863835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:58.863677 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" Oct 31 21:56:59.418823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:59.418758 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" podStartSLOduration=4.418738466 podStartE2EDuration="4.418738466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:56:56.721651755 +0000 UTC m=+2456.846758992" watchObservedRunningTime="2026-10-31 21:56:59.418738466 +0000 UTC m=+2459.543845703" Oct 31 21:56:59.419524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:56:59.419493 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk"] Oct 31 21:57:00.099121 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:00.099080 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:57:00.101514 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:00.101493 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 21:57:01.322228 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:57:01.322191 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:57:04.420404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.420362 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" podUID="dbd2e103-bd93-4960-b392-5c0e4d292da6" containerName="payload-processing" containerID="cri-o://af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801" gracePeriod=25 Oct 31 21:57:04.681648 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.681586 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:57:04.700108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.700081 4294 generic.go:356] "Generic (PLEG): container finished" podID="dbd2e103-bd93-4960-b392-5c0e4d292da6" containerID="af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801" exitCode=0 Oct 31 21:57:04.700264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.700123 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" Oct 31 21:57:04.700264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.700158 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" event={"ID":"dbd2e103-bd93-4960-b392-5c0e4d292da6","Type":"ContainerDied","Data":"af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801"} Oct 31 21:57:04.700264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.700208 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk" event={"ID":"dbd2e103-bd93-4960-b392-5c0e4d292da6","Type":"ContainerDied","Data":"9e94359109ccc043927f45c121b17d576d159b949c18ca21b817c4f7ce212651"} Oct 31 21:57:04.700264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.700231 4294 scope.go:122] "RemoveContainer" containerID="af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801" Oct 31 21:57:04.711338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.711318 4294 scope.go:122] "RemoveContainer" containerID="af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801" Oct 31 21:57:04.711648 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:57:04.711610 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801\": container with ID starting with af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801 not found: ID does not exist" containerID="af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801" Oct 31 21:57:04.711752 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.711660 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801"} err="failed to get container status \"af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801\": rpc error: code = NotFound desc = could not find container \"af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801\": container with ID starting with af1c7d44ff867a6c74c2f2a873ece23998bf9e274300a997c70c6f7387da6801 not found: ID does not exist" Oct 31 21:57:04.773108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.773078 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-plugins-config-volume\") pod \"dbd2e103-bd93-4960-b392-5c0e4d292da6\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " Oct 31 21:57:04.773256 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.773176 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-provider-credentials\") pod \"dbd2e103-bd93-4960-b392-5c0e4d292da6\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " Oct 31 21:57:04.773391 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.773371 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-kube-api-access-jc9tb\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-kube-api-access-jc9tb\") pod \"dbd2e103-bd93-4960-b392-5c0e4d292da6\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " Oct 31 21:57:04.773537 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.773518 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-routing-overlay-volume\") pod \"dbd2e103-bd93-4960-b392-5c0e4d292da6\" (UID: \"dbd2e103-bd93-4960-b392-5c0e4d292da6\") " Oct 31 21:57:04.773620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.773539 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-plugins-config-volume" pod "dbd2e103-bd93-4960-b392-5c0e4d292da6" (UID: "dbd2e103-bd93-4960-b392-5c0e4d292da6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:57:04.773846 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.773821 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-routing-overlay-volume" pod "dbd2e103-bd93-4960-b392-5c0e4d292da6" (UID: "dbd2e103-bd93-4960-b392-5c0e4d292da6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:57:04.774105 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.774088 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:04.774186 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.774109 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dbd2e103-bd93-4960-b392-5c0e4d292da6-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:04.775468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.775441 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-kube-api-access-jc9tb" pod "dbd2e103-bd93-4960-b392-5c0e4d292da6" (UID: "dbd2e103-bd93-4960-b392-5c0e4d292da6"). InnerVolumeSpecName "kube-api-access-jc9tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:57:04.775561 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.775516 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-provider-credentials" pod "dbd2e103-bd93-4960-b392-5c0e4d292da6" (UID: "dbd2e103-bd93-4960-b392-5c0e4d292da6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:57:04.875827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.875788 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:04.875827 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:04.875822 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc9tb\" (UniqueName: \"kubernetes.io/projected/dbd2e103-bd93-4960-b392-5c0e4d292da6-kube-api-access-jc9tb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:05.051600 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:05.051567 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk"] Oct 31 21:57:05.058009 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:05.057985 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788k64dk"] Oct 31 21:57:05.986743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:05.986707 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd2e103-bd93-4960-b392-5c0e4d292da6" path="/var/lib/kubelet/pods/dbd2e103-bd93-4960-b392-5c0e4d292da6/volumes" Oct 31 21:57:06.192468 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.192427 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647"] Oct 31 21:57:06.193131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.193109 4294 cpu_manager.go:414] "containerMap: removing container" podUID="dbd2e103-bd93-4960-b392-5c0e4d292da6" containerName="payload-processing" Oct 31 21:57:06.193131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.193132 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd2e103-bd93-4960-b392-5c0e4d292da6" containerName="payload-processing" Oct 31 21:57:06.193357 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.193343 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd2e103-bd93-4960-b392-5c0e4d292da6" containerName="payload-processing" Oct 31 21:57:06.198243 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.198220 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.217109 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.217084 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647"] Oct 31 21:57:06.223857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.223823 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"openshift-service-ca.crt\"" Oct 31 21:57:06.223857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.223852 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"kube-root-ca.crt\"" Oct 31 21:57:06.224096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.223926 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-56a6d4\"" Oct 31 21:57:06.224096 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.223984 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"routing-overlay\"" Oct 31 21:57:06.238626 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.238558 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"agc-cred-8162ebadb51a\"" Oct 31 21:57:06.238797 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.238775 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-e2e--34b55c0e-dockercfg-5bdlf\"" Oct 31 21:57:06.300323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.300272 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.300490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.300472 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.300593 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.300579 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7zgf\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-kube-api-access-t7zgf\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.300635 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.300613 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.401600 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.401569 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7zgf\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-kube-api-access-t7zgf\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.401600 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.401603 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.401834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.401669 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.401834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.401822 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.402345 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.402321 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.402488 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.402467 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.404694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.404655 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.424705 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.424678 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7zgf\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-kube-api-access-t7zgf\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.512715 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.512645 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:06.823081 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:06.823045 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647"] Oct 31 21:57:06.824801 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:57:06.824776 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87036d13_43a4_4bfa_9050_a9e27dc16be3.slice/crio-ecebb74e1a0630fc34c6dc17642815be623f644569e5a94bdfd6411d887e4106 WatchSource:0}: Error finding container ecebb74e1a0630fc34c6dc17642815be623f644569e5a94bdfd6411d887e4106: Status 404 returned error can't find the container with id ecebb74e1a0630fc34c6dc17642815be623f644569e5a94bdfd6411d887e4106 Oct 31 21:57:07.643042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:07.643014 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:57:07.725084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:07.725044 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" event={"ID":"87036d13-43a4-4bfa-9050-a9e27dc16be3","Type":"ContainerStarted","Data":"0f156f9b54765aa01996cd9ec6187eac76a08941613084921224ead9997c577c"} Oct 31 21:57:07.725272 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:07.725114 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" event={"ID":"87036d13-43a4-4bfa-9050-a9e27dc16be3","Type":"ContainerStarted","Data":"ecebb74e1a0630fc34c6dc17642815be623f644569e5a94bdfd6411d887e4106"} Oct 31 21:57:07.725272 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:07.725144 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:07.769629 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:07.769577 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" podStartSLOduration=1.769561941 podStartE2EDuration="1.769561941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:57:07.766671867 +0000 UTC m=+2467.891779098" watchObservedRunningTime="2026-10-31 21:57:07.769561941 +0000 UTC m=+2467.894669171" Oct 31 21:57:16.537543 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.537461 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647"] Oct 31 21:57:16.540925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.540896 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:16.571959 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.571922 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj"] Oct 31 21:57:16.576179 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.576160 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.598633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.598603 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj"] Oct 31 21:57:16.675716 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.675681 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.675900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.675852 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tfcc\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-kube-api-access-5tfcc\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.675900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.675889 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.676058 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.676036 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.681911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.681896 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"agc-cred-7aca9c7706de\"" Oct 31 21:57:16.777835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.777803 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tfcc\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-kube-api-access-5tfcc\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.777965 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.777851 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.777965 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.777913 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.778091 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.777970 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.778678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.778654 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.778765 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.778699 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.780439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.780415 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.801633 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.801577 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tfcc\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-kube-api-access-5tfcc\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:16.891343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:16.891294 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:17.195732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:17.195703 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj"] Oct 31 21:57:17.196701 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:57:17.196669 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f68e3c_754d_4995_a177_759a637337bd.slice/crio-8f26eb7a4955cbea8b97ac6f63b947331dd29956a0b8e72de40918aa9e895143 WatchSource:0}: Error finding container 8f26eb7a4955cbea8b97ac6f63b947331dd29956a0b8e72de40918aa9e895143: Status 404 returned error can't find the container with id 8f26eb7a4955cbea8b97ac6f63b947331dd29956a0b8e72de40918aa9e895143 Oct 31 21:57:17.799099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:17.799059 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" event={"ID":"e0f68e3c-754d-4995-a177-759a637337bd","Type":"ContainerStarted","Data":"e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8"} Oct 31 21:57:17.799646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:17.799126 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" event={"ID":"e0f68e3c-754d-4995-a177-759a637337bd","Type":"ContainerStarted","Data":"8f26eb7a4955cbea8b97ac6f63b947331dd29956a0b8e72de40918aa9e895143"} Oct 31 21:57:17.799646 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:17.799251 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:17.840724 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:17.840676 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" podStartSLOduration=1.840662611 podStartE2EDuration="1.840662611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:57:17.83982384 +0000 UTC m=+2477.964931065" watchObservedRunningTime="2026-10-31 21:57:17.840662611 +0000 UTC m=+2477.965769857" Oct 31 21:57:19.707912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:19.707877 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82"] Oct 31 21:57:19.720238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:19.720201 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f1128d-dkt82"] Oct 31 21:57:19.985559 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:19.985529 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009e5e96-81df-48f0-8701-4125535b8566" path="/var/lib/kubelet/pods/009e5e96-81df-48f0-8701-4125535b8566/volumes" Oct 31 21:57:21.538665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.538624 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" podUID="87036d13-43a4-4bfa-9050-a9e27dc16be3" containerName="payload-processing" containerID="cri-o://0f156f9b54765aa01996cd9ec6187eac76a08941613084921224ead9997c577c" gracePeriod=25 Oct 31 21:57:21.831431 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.831403 4294 generic.go:356] "Generic (PLEG): container finished" podID="87036d13-43a4-4bfa-9050-a9e27dc16be3" containerID="0f156f9b54765aa01996cd9ec6187eac76a08941613084921224ead9997c577c" exitCode=0 Oct 31 21:57:21.831575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.831487 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" event={"ID":"87036d13-43a4-4bfa-9050-a9e27dc16be3","Type":"ContainerDied","Data":"0f156f9b54765aa01996cd9ec6187eac76a08941613084921224ead9997c577c"} Oct 31 21:57:21.831575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.831551 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" event={"ID":"87036d13-43a4-4bfa-9050-a9e27dc16be3","Type":"ContainerDied","Data":"ecebb74e1a0630fc34c6dc17642815be623f644569e5a94bdfd6411d887e4106"} Oct 31 21:57:21.831652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.831575 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecebb74e1a0630fc34c6dc17642815be623f644569e5a94bdfd6411d887e4106" Oct 31 21:57:21.850259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.850238 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:21.951346 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.951317 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-provider-credentials\") pod \"87036d13-43a4-4bfa-9050-a9e27dc16be3\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " Oct 31 21:57:21.951511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.951375 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-kube-api-access-t7zgf\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-kube-api-access-t7zgf\") pod \"87036d13-43a4-4bfa-9050-a9e27dc16be3\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " Oct 31 21:57:21.951511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.951497 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-routing-overlay-volume\") pod \"87036d13-43a4-4bfa-9050-a9e27dc16be3\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " Oct 31 21:57:21.951588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.951524 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-plugins-config-volume\") pod \"87036d13-43a4-4bfa-9050-a9e27dc16be3\" (UID: \"87036d13-43a4-4bfa-9050-a9e27dc16be3\") " Oct 31 21:57:21.951843 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.951815 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-routing-overlay-volume" pod "87036d13-43a4-4bfa-9050-a9e27dc16be3" (UID: "87036d13-43a4-4bfa-9050-a9e27dc16be3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:57:21.952007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.951913 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-plugins-config-volume" pod "87036d13-43a4-4bfa-9050-a9e27dc16be3" (UID: "87036d13-43a4-4bfa-9050-a9e27dc16be3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:57:21.952170 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.952154 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:21.952234 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.952171 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87036d13-43a4-4bfa-9050-a9e27dc16be3-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:21.953492 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.953470 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-kube-api-access-t7zgf" pod "87036d13-43a4-4bfa-9050-a9e27dc16be3" (UID: "87036d13-43a4-4bfa-9050-a9e27dc16be3"). InnerVolumeSpecName "kube-api-access-t7zgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:57:21.953582 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:21.953518 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-provider-credentials" pod "87036d13-43a4-4bfa-9050-a9e27dc16be3" (UID: "87036d13-43a4-4bfa-9050-a9e27dc16be3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:57:22.053473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:22.053434 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:22.053473 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:22.053469 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7zgf\" (UniqueName: \"kubernetes.io/projected/87036d13-43a4-4bfa-9050-a9e27dc16be3-kube-api-access-t7zgf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:22.838067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:22.838037 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647" Oct 31 21:57:22.890470 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:22.890442 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647"] Oct 31 21:57:22.902980 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:22.902954 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97jn647"] Oct 31 21:57:23.985821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:23.985789 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87036d13-43a4-4bfa-9050-a9e27dc16be3" path="/var/lib/kubelet/pods/87036d13-43a4-4bfa-9050-a9e27dc16be3/volumes" Oct 31 21:57:27.132691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:27.132655 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc"] Oct 31 21:57:27.133177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:27.132919 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" containerID="cri-o://7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608" gracePeriod=30 Oct 31 21:57:28.811072 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:28.811044 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:57:31.311173 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:57:31.311140 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:57:32.287416 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.287386 4294 patch_prober.go:29] interesting pod/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 31 21:57:32.287580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.287458 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 31 21:57:32.414671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.414602 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:57:32.522884 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.522845 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-podinfo\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523060 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.522939 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4611dd89-b726-40e0-a371-9ca273fb0b8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4611dd89-b726-40e0-a371-9ca273fb0b8c-istiod-ca-cert\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523060 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523040 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-envoy\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523081 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-credential-socket\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523112 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-certs\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523143 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-socket\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523179 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-kube-api-access-2pp6v\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-kube-api-access-2pp6v\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523421 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523247 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-token\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-token\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523421 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523313 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-data\") pod \"4611dd89-b726-40e0-a371-9ca273fb0b8c\" (UID: \"4611dd89-b726-40e0-a371-9ca273fb0b8c\") " Oct 31 21:57:32.523539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523433 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-certs" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:57:32.523539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523462 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4611dd89-b726-40e0-a371-9ca273fb0b8c-istiod-ca-cert" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:57:32.523539 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523462 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-credential-socket" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:57:32.523703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523675 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-data" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:57:32.523703 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.523690 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-socket" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:57:32.524028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.524004 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4611dd89-b726-40e0-a371-9ca273fb0b8c-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.524028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.524034 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.524028 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.524049 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.524343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.524062 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.524343 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.524074 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.525754 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.525726 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-podinfo" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 21:57:32.525847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.525767 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-kube-api-access-2pp6v" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "kube-api-access-2pp6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:57:32.525847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.525786 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-envoy" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 21:57:32.527637 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.527614 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-token" pod "4611dd89-b726-40e0-a371-9ca273fb0b8c" (UID: "4611dd89-b726-40e0-a371-9ca273fb0b8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:57:32.625792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.625761 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.625792 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.625790 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pp6v\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-kube-api-access-2pp6v\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.625984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.625801 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.625984 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.625810 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4611dd89-b726-40e0-a371-9ca273fb0b8c-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:57:32.907920 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.907885 4294 generic.go:356] "Generic (PLEG): container finished" podID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerID="7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608" exitCode=0 Oct 31 21:57:32.908094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.907937 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" Oct 31 21:57:32.908094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.907958 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" event={"ID":"4611dd89-b726-40e0-a371-9ca273fb0b8c","Type":"ContainerDied","Data":"7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608"} Oct 31 21:57:32.908094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.908040 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc" event={"ID":"4611dd89-b726-40e0-a371-9ca273fb0b8c","Type":"ContainerDied","Data":"37a638b6e677a5f24dafeecd0db01462bb96ded585aedca41cb08d6f9ac7ebc0"} Oct 31 21:57:32.908094 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.908062 4294 scope.go:122] "RemoveContainer" containerID="7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608" Oct 31 21:57:32.919026 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.919010 4294 scope.go:122] "RemoveContainer" containerID="e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da" Oct 31 21:57:32.927780 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.927761 4294 scope.go:122] "RemoveContainer" containerID="7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608" Oct 31 21:57:32.928048 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:57:32.928030 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608\": container with ID starting with 7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608 not found: ID does not exist" containerID="7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608" Oct 31 21:57:32.928095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.928056 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608"} err="failed to get container status \"7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608\": rpc error: code = NotFound desc = could not find container \"7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608\": container with ID starting with 7aa5071834a39a46b7f7f77b65b41093dd13a6914c8bf2cb10ef23784fbc0608 not found: ID does not exist" Oct 31 21:57:32.928095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.928074 4294 scope.go:122] "RemoveContainer" containerID="e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da" Oct 31 21:57:32.928335 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:57:32.928290 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da\": container with ID starting with e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da not found: ID does not exist" containerID="e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da" Oct 31 21:57:32.928453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.928330 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da"} err="failed to get container status \"e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da\": rpc error: code = NotFound desc = could not find container \"e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da\": container with ID starting with e0a70f1db69f14872ef97a09292335ad13d078e674abdccbf2cc9360df0238da not found: ID does not exist" Oct 31 21:57:32.964551 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.964500 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc"] Oct 31 21:57:32.968713 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:32.968687 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-f1128d-openshift-default-7b78c96947-j7fmc"] Oct 31 21:57:33.987217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:33.987182 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" path="/var/lib/kubelet/pods/4611dd89-b726-40e0-a371-9ca273fb0b8c/volumes" Oct 31 21:57:35.283149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283113 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b"] Oct 31 21:57:35.283767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283753 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" Oct 31 21:57:35.283813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283768 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" Oct 31 21:57:35.283813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283794 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" Oct 31 21:57:35.283813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283799 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" Oct 31 21:57:35.283813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283810 4294 cpu_manager.go:414] "containerMap: removing container" podUID="87036d13-43a4-4bfa-9050-a9e27dc16be3" containerName="payload-processing" Oct 31 21:57:35.283944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283816 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87036d13-43a4-4bfa-9050-a9e27dc16be3" containerName="payload-processing" Oct 31 21:57:35.283944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283908 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" Oct 31 21:57:35.283944 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.283918 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="87036d13-43a4-4bfa-9050-a9e27dc16be3" containerName="payload-processing" Oct 31 21:57:35.288779 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.288758 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.305145 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.305122 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b"] Oct 31 21:57:35.312937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.312893 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-4275b9\"" Oct 31 21:57:35.313112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.313002 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-4275b9-dockercfg-xwb27\"" Oct 31 21:57:35.372696 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.372665 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh"] Oct 31 21:57:35.373367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.373353 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4611dd89-b726-40e0-a371-9ca273fb0b8c" containerName="istio-proxy" Oct 31 21:57:35.376821 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.376801 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.382038 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.382015 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmp4\" (UniqueName: \"kubernetes.io/projected/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-kube-api-access-kbmp4\") pod \"payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.382171 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.382160 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.395385 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.395360 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh"] Oct 31 21:57:35.513323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.493370 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmp4\" (UniqueName: \"kubernetes.io/projected/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-kube-api-access-kbmp4\") pod \"payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.513665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.513633 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbsfd\" (UniqueName: \"kubernetes.io/projected/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-kube-api-access-zbsfd\") pod \"payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.513947 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.513929 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.514071 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.514056 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.537511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.537433 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.569050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.569017 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmp4\" (UniqueName: \"kubernetes.io/projected/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-kube-api-access-kbmp4\") pod \"payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.602961 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.602663 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:35.615598 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.615554 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbsfd\" (UniqueName: \"kubernetes.io/projected/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-kube-api-access-zbsfd\") pod \"payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.615796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.615761 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.617064 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.617037 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.648078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.648010 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbsfd\" (UniqueName: \"kubernetes.io/projected/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-kube-api-access-zbsfd\") pod \"payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.691763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.691726 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:35.958025 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.957994 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b"] Oct 31 21:57:35.958579 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:57:35.958553 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa8d876_0d59_4a3b_a51c_a6c695a88aea.slice/crio-823c29957bcbcb9247099815b9b7f37e769b59e1f2a00fa813c614276b1e179c WatchSource:0}: Error finding container 823c29957bcbcb9247099815b9b7f37e769b59e1f2a00fa813c614276b1e179c: Status 404 returned error can't find the container with id 823c29957bcbcb9247099815b9b7f37e769b59e1f2a00fa813c614276b1e179c Oct 31 21:57:35.967245 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:35.967220 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" event={"ID":"1fa8d876-0d59-4a3b-a51c-a6c695a88aea","Type":"ContainerStarted","Data":"823c29957bcbcb9247099815b9b7f37e769b59e1f2a00fa813c614276b1e179c"} Oct 31 21:57:36.035796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:36.035769 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh"] Oct 31 21:57:36.036529 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:57:36.036453 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29aef6e9_a20a_4ffd_af1a_2ddb84ace701.slice/crio-034981e22417fc938d71a1997ef08f28a9bf369e9462805d7014fe4dfd6d312e WatchSource:0}: Error finding container 034981e22417fc938d71a1997ef08f28a9bf369e9462805d7014fe4dfd6d312e: Status 404 returned error can't find the container with id 034981e22417fc938d71a1997ef08f28a9bf369e9462805d7014fe4dfd6d312e Oct 31 21:57:37.006326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:37.004170 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" event={"ID":"29aef6e9-a20a-4ffd-af1a-2ddb84ace701","Type":"ContainerStarted","Data":"ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6"} Oct 31 21:57:37.006326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:37.004238 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" event={"ID":"29aef6e9-a20a-4ffd-af1a-2ddb84ace701","Type":"ContainerStarted","Data":"034981e22417fc938d71a1997ef08f28a9bf369e9462805d7014fe4dfd6d312e"} Oct 31 21:57:37.006326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:37.004608 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:37.007213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:37.007182 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" event={"ID":"1fa8d876-0d59-4a3b-a51c-a6c695a88aea","Type":"ContainerStarted","Data":"9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8"} Oct 31 21:57:37.007389 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:37.007253 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:37.057583 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:37.057516 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" podStartSLOduration=2.05749659 podStartE2EDuration="2.05749659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:57:37.045016716 +0000 UTC m=+2497.170123947" watchObservedRunningTime="2026-10-31 21:57:37.05749659 +0000 UTC m=+2497.182603823" Oct 31 21:57:37.098268 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:37.098191 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" podStartSLOduration=2.098166863 podStartE2EDuration="2.098166863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:57:37.085711643 +0000 UTC m=+2497.210818874" watchObservedRunningTime="2026-10-31 21:57:37.098166863 +0000 UTC m=+2497.223274093" Oct 31 21:57:44.982978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:44.982942 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5"] Oct 31 21:57:45.004878 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.004837 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.012786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.007615 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5"] Oct 31 21:57:45.020112 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.020089 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"openshift-service-ca.crt\"" Oct 31 21:57:45.031915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.031880 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"kube-root-ca.crt\"" Oct 31 21:57:45.031915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.031905 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"routing-overlay\"" Oct 31 21:57:45.032127 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.031962 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"payload-processing-external-model-plugins-e2e-extmod-b-c34cbd\"" Oct 31 21:57:45.087761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.087725 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"agc-cred-3d39da95e8af\"" Oct 31 21:57:45.087761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.087732 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"agc-cred-0185341c205f\"" Oct 31 21:57:45.087761 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.087759 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-c34cbd\"/\"payload-processing-external-model-e2e--bb6f7797-dockercfg-6p5zs\"" Oct 31 21:57:45.088888 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.088863 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.089030 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.088932 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.089078 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.089042 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95pb7\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-kube-api-access-95pb7\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.089204 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.089186 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.190592 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.190555 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.190803 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.190703 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95pb7\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-kube-api-access-95pb7\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.190879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.190811 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.190879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.190872 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.191500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.191462 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.191683 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.191663 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.193124 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.193099 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.214998 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.214967 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95pb7\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-kube-api-access-95pb7\") pod \"payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.342616 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.342536 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:45.671210 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.671109 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5"] Oct 31 21:57:45.671836 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:57:45.671799 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e88aa2_73ae_4a96_b327_9aeec9d3e1c8.slice/crio-4b418f9dc89a6d5d260553d7ffe3648cb6dbd6ae904ca7bd232faddd9c745acf WatchSource:0}: Error finding container 4b418f9dc89a6d5d260553d7ffe3648cb6dbd6ae904ca7bd232faddd9c745acf: Status 404 returned error can't find the container with id 4b418f9dc89a6d5d260553d7ffe3648cb6dbd6ae904ca7bd232faddd9c745acf Oct 31 21:57:45.756426 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.756389 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": read tcp 10.134.0.2:52518->10.134.0.201:15021: read: connection reset by peer" start-of-body= Oct 31 21:57:45.756665 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:45.756458 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": read tcp 10.134.0.2:52518->10.134.0.201:15021: read: connection reset by peer" Oct 31 21:57:46.073369 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.073291 4294 generic.go:356] "Generic (PLEG): container finished" podID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerID="4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b" exitCode=0 Oct 31 21:57:46.073806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.073369 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" event={"ID":"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573","Type":"ContainerDied","Data":"4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b"} Oct 31 21:57:46.073876 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.073823 4294 scope.go:122] "RemoveContainer" containerID="4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b" Oct 31 21:57:46.075153 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.075131 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" event={"ID":"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8","Type":"ContainerStarted","Data":"28f6a0e5a828c3407e55936a444313567be5a137d6b9e5d57b959fe535d51bfa"} Oct 31 21:57:46.075259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.075174 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" event={"ID":"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8","Type":"ContainerStarted","Data":"4b418f9dc89a6d5d260553d7ffe3648cb6dbd6ae904ca7bd232faddd9c745acf"} Oct 31 21:57:46.075259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.075196 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:57:46.075673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.075646 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:57:46.075723 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.075711 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:57:46.075758 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.075746 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:57:46.163594 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:46.163550 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" podStartSLOduration=2.163536447 podStartE2EDuration="2.163536447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:57:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:57:46.162725701 +0000 UTC m=+2506.287832933" watchObservedRunningTime="2026-10-31 21:57:46.163536447 +0000 UTC m=+2506.288643677" Oct 31 21:57:47.090775 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:47.090733 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" event={"ID":"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573","Type":"ContainerStarted","Data":"2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a"} Oct 31 21:57:47.552772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:47.552733 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:57:47.552963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:47.552933 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 21:57:47.553011 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:47.552975 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 21:57:48.018417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:48.018387 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 21:57:48.018786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:48.018767 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 21:57:48.552973 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:48.552942 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 21:57:48.553394 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:48.552991 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 21:57:49.553139 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:49.553106 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 21:57:49.553569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:49.553153 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 21:57:50.552997 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:50.552962 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 21:57:50.553177 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:50.553013 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 21:57:51.552926 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:51.552885 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 31 21:57:51.553115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:51.552941 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 31 21:57:52.557464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:52.557431 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:57:52.557885 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:52.557838 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:57:52.558579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:52.558559 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 21:57:57.093690 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:57:57.093652 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:58:01.310486 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:58:01.310450 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:58:01.957627 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:01.957588 4294 scope.go:122] "RemoveContainer" containerID="0bfdfb99ab1f69ceeb745bb0e33017b1e5d9cb075c32bb025d7bcac3ea1c16ed" Oct 31 21:58:01.968814 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:01.968792 4294 scope.go:122] "RemoveContainer" containerID="34dde9afd4b25262d641dd71cc4360c170edb6fc8d7ed4bb7a1aecc99bf4c043" Oct 31 21:58:01.978535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:01.978514 4294 scope.go:122] "RemoveContainer" containerID="68c696f600ad9b081020e62b90568b15a2d2a6e890498fa255a1eab5d966bc19" Oct 31 21:58:11.076617 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.076584 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt"] Oct 31 21:58:11.082614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.082594 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.099810 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.099784 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt"] Oct 31 21:58:11.106618 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.106593 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"agc-cred-da7a4611d93a\"" Oct 31 21:58:11.206789 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.206755 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.206963 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.206922 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.207006 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.206980 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rsz\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-kube-api-access-76rsz\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.207169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.207147 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.308015 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.307981 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.308217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.308040 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rsz\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-kube-api-access-76rsz\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.308217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.308124 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.308217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.308194 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.308768 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.308741 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.308912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.308836 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.310609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.310582 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.331508 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.331440 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rsz\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-kube-api-access-76rsz\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.397164 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.397137 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:11.700569 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:11.700541 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt"] Oct 31 21:58:11.701886 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:58:11.701861 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f7f141_2169_4d29_8c8a_0b272be18091.slice/crio-909cbb957192a7831d6e9fab7f3b499e7564ea7e2ce1f1ebdfa47bfdea0daae6 WatchSource:0}: Error finding container 909cbb957192a7831d6e9fab7f3b499e7564ea7e2ce1f1ebdfa47bfdea0daae6: Status 404 returned error can't find the container with id 909cbb957192a7831d6e9fab7f3b499e7564ea7e2ce1f1ebdfa47bfdea0daae6 Oct 31 21:58:12.292137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:12.292095 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" event={"ID":"c2f7f141-2169-4d29-8c8a-0b272be18091","Type":"ContainerStarted","Data":"1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4"} Oct 31 21:58:12.292575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:12.292159 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" event={"ID":"c2f7f141-2169-4d29-8c8a-0b272be18091","Type":"ContainerStarted","Data":"909cbb957192a7831d6e9fab7f3b499e7564ea7e2ce1f1ebdfa47bfdea0daae6"} Oct 31 21:58:12.292575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:12.292184 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:12.344868 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:12.344819 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" podStartSLOduration=1.344804589 podStartE2EDuration="1.344804589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:58:12.344049694 +0000 UTC m=+2532.469156925" watchObservedRunningTime="2026-10-31 21:58:12.344804589 +0000 UTC m=+2532.469911819" Oct 31 21:58:18.289634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.289602 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz"] Oct 31 21:58:18.294027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.294010 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.310476 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.310455 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:58:18.310709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.310688 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz"] Oct 31 21:58:18.321455 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.321430 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:58:18.416475 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.416446 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdm46\" (UniqueName: \"kubernetes.io/projected/8d1d4569-1867-4943-ba5e-568e8c281339-kube-api-access-mdm46\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.416661 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.416645 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.416714 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.416705 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.518532 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.518500 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.518702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.518595 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.518702 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.518679 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdm46\" (UniqueName: \"kubernetes.io/projected/8d1d4569-1867-4943-ba5e-568e8c281339-kube-api-access-mdm46\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.519272 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.519248 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.520794 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.520772 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.544039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.543986 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdm46\" (UniqueName: \"kubernetes.io/projected/8d1d4569-1867-4943-ba5e-568e8c281339-kube-api-access-mdm46\") pod \"maas-api-revoke-sub-4ca318a4-ktspz\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.612486 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.612455 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:18.914824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:18.914794 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz"] Oct 31 21:58:18.915723 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:58:18.915697 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1d4569_1867_4943_ba5e_568e8c281339.slice/crio-5ed4da2b0510537ebc12eac47b9576f03628b1ce26a7c68b7545ffa39621964c WatchSource:0}: Error finding container 5ed4da2b0510537ebc12eac47b9576f03628b1ce26a7c68b7545ffa39621964c: Status 404 returned error can't find the container with id 5ed4da2b0510537ebc12eac47b9576f03628b1ce26a7c68b7545ffa39621964c Oct 31 21:58:19.348937 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:19.348893 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" event={"ID":"8d1d4569-1867-4943-ba5e-568e8c281339","Type":"ContainerStarted","Data":"28b042543fc93643328cdee1a6be5a96add2a134e632586a24d1a3686514ba03"} Oct 31 21:58:19.349328 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:19.348979 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" event={"ID":"8d1d4569-1867-4943-ba5e-568e8c281339","Type":"ContainerStarted","Data":"5ed4da2b0510537ebc12eac47b9576f03628b1ce26a7c68b7545ffa39621964c"} Oct 31 21:58:19.395812 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:19.395757 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" podStartSLOduration=1.395743542 podStartE2EDuration="1.395743542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:58:19.392852164 +0000 UTC m=+2539.517959395" watchObservedRunningTime="2026-10-31 21:58:19.395743542 +0000 UTC m=+2539.520850772" Oct 31 21:58:20.358377 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:20.358344 4294 generic.go:356] "Generic (PLEG): container finished" podID="8d1d4569-1867-4943-ba5e-568e8c281339" containerID="28b042543fc93643328cdee1a6be5a96add2a134e632586a24d1a3686514ba03" exitCode=0 Oct 31 21:58:20.358899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:20.358457 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" event={"ID":"8d1d4569-1867-4943-ba5e-568e8c281339","Type":"ContainerDied","Data":"28b042543fc93643328cdee1a6be5a96add2a134e632586a24d1a3686514ba03"} Oct 31 21:58:21.513360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.513334 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:21.560727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.560700 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1d4569-1867-4943-ba5e-568e8c281339-kube-api-access-mdm46\" (UniqueName: \"kubernetes.io/projected/8d1d4569-1867-4943-ba5e-568e8c281339-kube-api-access-mdm46\") pod \"8d1d4569-1867-4943-ba5e-568e8c281339\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " Oct 31 21:58:21.560988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.560967 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-xks-ca\") pod \"8d1d4569-1867-4943-ba5e-568e8c281339\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " Oct 31 21:58:21.561092 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.561076 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-openshift-ca\") pod \"8d1d4569-1867-4943-ba5e-568e8c281339\" (UID: \"8d1d4569-1867-4943-ba5e-568e8c281339\") " Oct 31 21:58:21.561796 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.561763 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-openshift-ca" pod "8d1d4569-1867-4943-ba5e-568e8c281339" (UID: "8d1d4569-1867-4943-ba5e-568e8c281339"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:21.563899 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.563875 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1d4569-1867-4943-ba5e-568e8c281339-kube-api-access-mdm46" pod "8d1d4569-1867-4943-ba5e-568e8c281339" (UID: "8d1d4569-1867-4943-ba5e-568e8c281339"). InnerVolumeSpecName "kube-api-access-mdm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:21.564155 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.564132 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-xks-ca" pod "8d1d4569-1867-4943-ba5e-568e8c281339" (UID: "8d1d4569-1867-4943-ba5e-568e8c281339"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:58:21.663140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.663062 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdm46\" (UniqueName: \"kubernetes.io/projected/8d1d4569-1867-4943-ba5e-568e8c281339-kube-api-access-mdm46\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:21.663140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.663090 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:21.663140 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:21.663109 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d1d4569-1867-4943-ba5e-568e8c281339-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:22.372362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:22.372268 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" Oct 31 21:58:22.372362 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:22.372318 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz" event={"ID":"8d1d4569-1867-4943-ba5e-568e8c281339","Type":"ContainerDied","Data":"5ed4da2b0510537ebc12eac47b9576f03628b1ce26a7c68b7545ffa39621964c"} Oct 31 21:58:22.372534 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:22.372372 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ed4da2b0510537ebc12eac47b9576f03628b1ce26a7c68b7545ffa39621964c" Oct 31 21:58:23.301941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:23.301911 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:23.393043 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:23.393010 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp"] Oct 31 21:58:24.228051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:24.228012 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5"] Oct 31 21:58:28.394235 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.394180 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" podUID="4259b89b-6819-4f46-a876-451517fbe9aa" containerName="payload-processing" containerID="cri-o://2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952" gracePeriod=25 Oct 31 21:58:28.665619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.665592 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:58:28.685895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.685867 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-kube-api-access-q8gc2\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-kube-api-access-q8gc2\") pod \"4259b89b-6819-4f46-a876-451517fbe9aa\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " Oct 31 21:58:28.686050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.685969 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-routing-overlay-volume\") pod \"4259b89b-6819-4f46-a876-451517fbe9aa\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " Oct 31 21:58:28.686117 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.686063 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-plugins-config-volume\") pod \"4259b89b-6819-4f46-a876-451517fbe9aa\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " Oct 31 21:58:28.686183 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.686129 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-provider-credentials\") pod \"4259b89b-6819-4f46-a876-451517fbe9aa\" (UID: \"4259b89b-6819-4f46-a876-451517fbe9aa\") " Oct 31 21:58:28.686360 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.686335 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-routing-overlay-volume" pod "4259b89b-6819-4f46-a876-451517fbe9aa" (UID: "4259b89b-6819-4f46-a876-451517fbe9aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:28.686498 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.686478 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-plugins-config-volume" pod "4259b89b-6819-4f46-a876-451517fbe9aa" (UID: "4259b89b-6819-4f46-a876-451517fbe9aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:28.686788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.686774 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:28.686829 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.686791 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4259b89b-6819-4f46-a876-451517fbe9aa-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:28.688042 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.688020 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-provider-credentials" pod "4259b89b-6819-4f46-a876-451517fbe9aa" (UID: "4259b89b-6819-4f46-a876-451517fbe9aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:28.688148 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.688097 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-kube-api-access-q8gc2" pod "4259b89b-6819-4f46-a876-451517fbe9aa" (UID: "4259b89b-6819-4f46-a876-451517fbe9aa"). InnerVolumeSpecName "kube-api-access-q8gc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:28.788452 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.788415 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gc2\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-kube-api-access-q8gc2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:28.788452 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:28.788446 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4259b89b-6819-4f46-a876-451517fbe9aa-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:29.228712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.228667 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" podUID="63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" containerName="payload-processing" containerID="cri-o://28f6a0e5a828c3407e55936a444313567be5a137d6b9e5d57b959fe535d51bfa" gracePeriod=25 Oct 31 21:58:29.323931 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.323897 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd"] Oct 31 21:58:29.324535 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.324521 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8d1d4569-1867-4943-ba5e-568e8c281339" containerName="revoke-keys" Oct 31 21:58:29.324590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.324535 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1d4569-1867-4943-ba5e-568e8c281339" containerName="revoke-keys" Oct 31 21:58:29.324590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.324552 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4259b89b-6819-4f46-a876-451517fbe9aa" containerName="payload-processing" Oct 31 21:58:29.324590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.324557 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4259b89b-6819-4f46-a876-451517fbe9aa" containerName="payload-processing" Oct 31 21:58:29.324689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.324663 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1d4569-1867-4943-ba5e-568e8c281339" containerName="revoke-keys" Oct 31 21:58:29.324689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.324674 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4259b89b-6819-4f46-a876-451517fbe9aa" containerName="payload-processing" Oct 31 21:58:29.328294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.328275 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.347063 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.347030 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd"] Oct 31 21:58:29.354969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.354919 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:58:29.354969 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.354942 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:58:29.397621 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.397583 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpswd\" (UniqueName: \"kubernetes.io/projected/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-kube-api-access-xpswd\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.397996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.397768 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.397996 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.397825 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.490200 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.490168 4294 generic.go:356] "Generic (PLEG): container finished" podID="4259b89b-6819-4f46-a876-451517fbe9aa" containerID="2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952" exitCode=0 Oct 31 21:58:29.490501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.490214 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" Oct 31 21:58:29.490501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.490256 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" event={"ID":"4259b89b-6819-4f46-a876-451517fbe9aa","Type":"ContainerDied","Data":"2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952"} Oct 31 21:58:29.490501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.490381 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp" event={"ID":"4259b89b-6819-4f46-a876-451517fbe9aa","Type":"ContainerDied","Data":"e43840f1dc99955f237d49b8b5b211605652a1b74f25af158ed7cc46bd2864b3"} Oct 31 21:58:29.490501 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.490424 4294 scope.go:122] "RemoveContainer" containerID="2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952" Oct 31 21:58:29.492175 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.492147 4294 generic.go:356] "Generic (PLEG): container finished" podID="63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" containerID="28f6a0e5a828c3407e55936a444313567be5a137d6b9e5d57b959fe535d51bfa" exitCode=0 Oct 31 21:58:29.492287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.492202 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" event={"ID":"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8","Type":"ContainerDied","Data":"28f6a0e5a828c3407e55936a444313567be5a137d6b9e5d57b959fe535d51bfa"} Oct 31 21:58:29.499595 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.499566 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpswd\" (UniqueName: \"kubernetes.io/projected/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-kube-api-access-xpswd\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.499770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.499750 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.499837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.499820 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.500361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.500341 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.501192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.501055 4294 scope.go:122] "RemoveContainer" containerID="2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952" Oct 31 21:58:29.502400 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.502368 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.503411 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:58:29.503372 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952\": container with ID starting with 2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952 not found: ID does not exist" containerID="2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952" Oct 31 21:58:29.503518 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.503423 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952"} err="failed to get container status \"2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952\": rpc error: code = NotFound desc = could not find container \"2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952\": container with ID starting with 2e0bec92189fd6ee57f7d22303c5c91340bcc45c90d64de65c2225bcf4be2952 not found: ID does not exist" Oct 31 21:58:29.524602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.524574 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpswd\" (UniqueName: \"kubernetes.io/projected/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-kube-api-access-xpswd\") pod \"maas-api-revoke-sub-3ad9afa0-hkcvd\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.549114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.549087 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:58:29.550376 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.550289 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp"] Oct 31 21:58:29.553874 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.553853 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-6ff9cfpkp"] Oct 31 21:58:29.601689 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.601655 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-plugins-config-volume\") pod \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " Oct 31 21:58:29.601879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.601770 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-routing-overlay-volume\") pod \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " Oct 31 21:58:29.601879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.601816 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-provider-credentials\") pod \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " Oct 31 21:58:29.601879 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.601855 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-kube-api-access-95pb7\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-kube-api-access-95pb7\") pod \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\" (UID: \"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8\") " Oct 31 21:58:29.602349 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.602148 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-plugins-config-volume" pod "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" (UID: "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:29.602897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.602623 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-routing-overlay-volume" pod "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" (UID: "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:29.602897 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.602717 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:29.605223 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.605197 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-kube-api-access-95pb7" pod "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" (UID: "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8"). InnerVolumeSpecName "kube-api-access-95pb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:29.619762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.619719 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-provider-credentials" pod "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" (UID: "63e88aa2-73ae-4a96-b327-9aeec9d3e1c8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:29.642371 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.642341 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:29.714084 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.714049 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:29.714217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.714089 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:29.714217 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.714105 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95pb7\" (UniqueName: \"kubernetes.io/projected/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8-kube-api-access-95pb7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:29.960276 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.960244 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd"] Oct 31 21:58:29.961439 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:58:29.961413 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37fd1f9_fa35_48f3_82c8_078ecd00d47f.slice/crio-10f5d3ba2e06c70773b0de7d37687d9eb9ba36ce7828f6981b8bfb7f82b5e83e WatchSource:0}: Error finding container 10f5d3ba2e06c70773b0de7d37687d9eb9ba36ce7828f6981b8bfb7f82b5e83e: Status 404 returned error can't find the container with id 10f5d3ba2e06c70773b0de7d37687d9eb9ba36ce7828f6981b8bfb7f82b5e83e Oct 31 21:58:29.987678 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:29.987654 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4259b89b-6819-4f46-a876-451517fbe9aa" path="/var/lib/kubelet/pods/4259b89b-6819-4f46-a876-451517fbe9aa/volumes" Oct 31 21:58:30.501699 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.501666 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" event={"ID":"c37fd1f9-fa35-48f3-82c8-078ecd00d47f","Type":"ContainerStarted","Data":"2c5668e6a0f5a2efcbcf7a4c52a09698dc7fc78e9838a76159699ee7b25e117e"} Oct 31 21:58:30.502151 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.501716 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" event={"ID":"c37fd1f9-fa35-48f3-82c8-078ecd00d47f","Type":"ContainerStarted","Data":"10f5d3ba2e06c70773b0de7d37687d9eb9ba36ce7828f6981b8bfb7f82b5e83e"} Oct 31 21:58:30.503023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.503000 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" event={"ID":"63e88aa2-73ae-4a96-b327-9aeec9d3e1c8","Type":"ContainerDied","Data":"4b418f9dc89a6d5d260553d7ffe3648cb6dbd6ae904ca7bd232faddd9c745acf"} Oct 31 21:58:30.503149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.503022 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5" Oct 31 21:58:30.503149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.503047 4294 scope.go:122] "RemoveContainer" containerID="28f6a0e5a828c3407e55936a444313567be5a137d6b9e5d57b959fe535d51bfa" Oct 31 21:58:30.544571 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.544478 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" podStartSLOduration=1.544464624 podStartE2EDuration="1.544464624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:58:30.543847764 +0000 UTC m=+2550.668954995" watchObservedRunningTime="2026-10-31 21:58:30.544464624 +0000 UTC m=+2550.669571854" Oct 31 21:58:30.585786 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.585756 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5"] Oct 31 21:58:30.598272 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:30.594929 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-c34cbd/payload-processing-external-model-e2e-extmod-b-c34cbd-5788hrsp5"] Oct 31 21:58:31.311574 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:58:31.311539 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:58:31.511694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:31.511661 4294 generic.go:356] "Generic (PLEG): container finished" podID="c37fd1f9-fa35-48f3-82c8-078ecd00d47f" containerID="2c5668e6a0f5a2efcbcf7a4c52a09698dc7fc78e9838a76159699ee7b25e117e" exitCode=0 Oct 31 21:58:31.512100 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:31.511744 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" event={"ID":"c37fd1f9-fa35-48f3-82c8-078ecd00d47f","Type":"ContainerDied","Data":"2c5668e6a0f5a2efcbcf7a4c52a09698dc7fc78e9838a76159699ee7b25e117e"} Oct 31 21:58:31.985402 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:31.985364 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" path="/var/lib/kubelet/pods/63e88aa2-73ae-4a96-b327-9aeec9d3e1c8/volumes" Oct 31 21:58:32.676739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.676716 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:32.760453 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.760424 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-xks-ca\") pod \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " Oct 31 21:58:32.760634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.760471 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-kube-api-access-xpswd\" (UniqueName: \"kubernetes.io/projected/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-kube-api-access-xpswd\") pod \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " Oct 31 21:58:32.760634 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.760559 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-openshift-ca\") pod \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\" (UID: \"c37fd1f9-fa35-48f3-82c8-078ecd00d47f\") " Oct 31 21:58:32.761073 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.761039 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-openshift-ca" pod "c37fd1f9-fa35-48f3-82c8-078ecd00d47f" (UID: "c37fd1f9-fa35-48f3-82c8-078ecd00d47f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:32.761386 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.761364 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:32.762624 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.762601 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-xks-ca" pod "c37fd1f9-fa35-48f3-82c8-078ecd00d47f" (UID: "c37fd1f9-fa35-48f3-82c8-078ecd00d47f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:58:32.763023 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.763004 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-kube-api-access-xpswd" pod "c37fd1f9-fa35-48f3-82c8-078ecd00d47f" (UID: "c37fd1f9-fa35-48f3-82c8-078ecd00d47f"). InnerVolumeSpecName "kube-api-access-xpswd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:32.863040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.863010 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:32.863040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:32.863037 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpswd\" (UniqueName: \"kubernetes.io/projected/c37fd1f9-fa35-48f3-82c8-078ecd00d47f-kube-api-access-xpswd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:33.527590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:33.527555 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" Oct 31 21:58:33.527590 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:33.527566 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd" event={"ID":"c37fd1f9-fa35-48f3-82c8-078ecd00d47f","Type":"ContainerDied","Data":"10f5d3ba2e06c70773b0de7d37687d9eb9ba36ce7828f6981b8bfb7f82b5e83e"} Oct 31 21:58:33.527862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:33.527624 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10f5d3ba2e06c70773b0de7d37687d9eb9ba36ce7828f6981b8bfb7f82b5e83e" Oct 31 21:58:40.563036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.562997 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx"] Oct 31 21:58:40.563667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.563651 4294 cpu_manager.go:414] "containerMap: removing container" podUID="63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" containerName="payload-processing" Oct 31 21:58:40.563735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.563668 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" containerName="payload-processing" Oct 31 21:58:40.563735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.563681 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c37fd1f9-fa35-48f3-82c8-078ecd00d47f" containerName="revoke-keys" Oct 31 21:58:40.563735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.563687 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37fd1f9-fa35-48f3-82c8-078ecd00d47f" containerName="revoke-keys" Oct 31 21:58:40.563835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.563781 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="63e88aa2-73ae-4a96-b327-9aeec9d3e1c8" containerName="payload-processing" Oct 31 21:58:40.563835 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.563791 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37fd1f9-fa35-48f3-82c8-078ecd00d47f" containerName="revoke-keys" Oct 31 21:58:40.568467 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.568438 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.588288 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.588257 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx"] Oct 31 21:58:40.596039 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.596011 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"agc-cred-192c5e9d9ff4\"" Oct 31 21:58:40.703298 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.703261 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.703497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.703397 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.703497 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.703447 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv84n\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-kube-api-access-zv84n\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.703579 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.703512 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.804911 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.804874 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.805116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.804929 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.805116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.804956 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv84n\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-kube-api-access-zv84n\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.805116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.804973 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.805644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.805616 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.805762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.805717 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.807570 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.807538 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.829500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.829446 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv84n\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-kube-api-access-zv84n\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:40.887837 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:40.887803 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:41.220135 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.220105 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx"] Oct 31 21:58:41.221445 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:58:41.221417 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc977fc93_3845_44c2_a8b8_3bdafbcd2656.slice/crio-395959dc19ed6faa0ac46e0d06de39153da7dfd63868ce330328009c073ee568 WatchSource:0}: Error finding container 395959dc19ed6faa0ac46e0d06de39153da7dfd63868ce330328009c073ee568: Status 404 returned error can't find the container with id 395959dc19ed6faa0ac46e0d06de39153da7dfd63868ce330328009c073ee568 Oct 31 21:58:41.597614 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.597578 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" event={"ID":"c977fc93-3845-44c2-a8b8-3bdafbcd2656","Type":"ContainerStarted","Data":"872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc"} Oct 31 21:58:41.598192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.597636 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" event={"ID":"c977fc93-3845-44c2-a8b8-3bdafbcd2656","Type":"ContainerStarted","Data":"395959dc19ed6faa0ac46e0d06de39153da7dfd63868ce330328009c073ee568"} Oct 31 21:58:41.598192 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.597768 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:41.685320 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.679153 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" podStartSLOduration=1.679130861 podStartE2EDuration="1.679130861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:58:41.657738523 +0000 UTC m=+2561.782845768" watchObservedRunningTime="2026-10-31 21:58:41.679130861 +0000 UTC m=+2561.804238092" Oct 31 21:58:41.736011 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.735978 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf"] Oct 31 21:58:41.740427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.740409 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:41.756709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.756682 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:58:41.756845 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.756791 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:58:41.759040 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.758998 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf"] Oct 31 21:58:41.818733 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.818701 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl"] Oct 31 21:58:41.823034 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.823011 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:41.840199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.840175 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"routing-overlay\"" Oct 31 21:58:41.840406 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.840216 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"kube-root-ca.crt\"" Oct 31 21:58:41.840494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.840226 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-696f9a\"" Oct 31 21:58:41.840494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.840239 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"openshift-service-ca.crt\"" Oct 31 21:58:41.841338 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.841310 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl"] Oct 31 21:58:41.856681 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.856627 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"agc-cred-f5769543c4b0\"" Oct 31 21:58:41.856841 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.856822 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-e2e--956cda1a-dockercfg-pqvgf\"" Oct 31 21:58:41.921181 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.921154 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:41.921361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.921193 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:41.921361 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.921286 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:41.921479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.921367 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9rqq\" (UniqueName: \"kubernetes.io/projected/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-kube-api-access-t9rqq\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:41.921479 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.921414 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:41.921552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.921498 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nhsr\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-kube-api-access-6nhsr\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:41.921552 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:41.921517 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.026326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.025322 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:42.026326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.025392 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.026326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.025482 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.026326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.025550 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9rqq\" (UniqueName: \"kubernetes.io/projected/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-kube-api-access-t9rqq\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:42.026326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.025641 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:42.026326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.025747 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nhsr\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-kube-api-access-6nhsr\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.026326 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.025789 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.029654 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.029497 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:42.030987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.030811 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.046323 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.032194 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.052324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.048663 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.052324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.048709 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:42.066433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.060968 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9rqq\" (UniqueName: \"kubernetes.io/projected/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-kube-api-access-t9rqq\") pod \"maas-api-revoke-sub-9d2e706d-bz8pf\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:42.066433 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.065792 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nhsr\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-kube-api-access-6nhsr\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.140098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.140008 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.355180 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.355144 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:42.459372 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.459344 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl"] Oct 31 21:58:42.460514 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:58:42.460488 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1468cf_90bf_4368_9196_9346701a52d3.slice/crio-5ec60a29e7196556b7ec1e80d0c3b5c503adf5f3b83f22b943b26d08cc10d9b4 WatchSource:0}: Error finding container 5ec60a29e7196556b7ec1e80d0c3b5c503adf5f3b83f22b943b26d08cc10d9b4: Status 404 returned error can't find the container with id 5ec60a29e7196556b7ec1e80d0c3b5c503adf5f3b83f22b943b26d08cc10d9b4 Oct 31 21:58:42.606056 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.606019 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" event={"ID":"fc1468cf-90bf-4368-9196-9346701a52d3","Type":"ContainerStarted","Data":"9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532"} Oct 31 21:58:42.606528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.606079 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" event={"ID":"fc1468cf-90bf-4368-9196-9346701a52d3","Type":"ContainerStarted","Data":"5ec60a29e7196556b7ec1e80d0c3b5c503adf5f3b83f22b943b26d08cc10d9b4"} Oct 31 21:58:42.606528 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.606232 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:42.656097 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.656033 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" podStartSLOduration=1.656015249 podStartE2EDuration="1.656015249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:58:42.650922396 +0000 UTC m=+2562.776029626" watchObservedRunningTime="2026-10-31 21:58:42.656015249 +0000 UTC m=+2562.781122479" Oct 31 21:58:42.660014 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:58:42.659938 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827a8b47_7adf_4b07_be7c_fb4dc2bdb767.slice/crio-9ac9c555d8fa938319184e80540a5ac6548c518ce1a7ec82569d7b853d131004 WatchSource:0}: Error finding container 9ac9c555d8fa938319184e80540a5ac6548c518ce1a7ec82569d7b853d131004: Status 404 returned error can't find the container with id 9ac9c555d8fa938319184e80540a5ac6548c518ce1a7ec82569d7b853d131004 Oct 31 21:58:42.661076 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:42.661054 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf"] Oct 31 21:58:43.615450 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:43.615413 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" event={"ID":"827a8b47-7adf-4b07-be7c-fb4dc2bdb767","Type":"ContainerDied","Data":"2f908665369db95ff9af4fc735e5aca38fc962aa4abc6520f28114ff263b86a0"} Oct 31 21:58:43.615831 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:43.615499 4294 generic.go:356] "Generic (PLEG): container finished" podID="827a8b47-7adf-4b07-be7c-fb4dc2bdb767" containerID="2f908665369db95ff9af4fc735e5aca38fc962aa4abc6520f28114ff263b86a0" exitCode=0 Oct 31 21:58:43.616503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:43.616475 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" event={"ID":"827a8b47-7adf-4b07-be7c-fb4dc2bdb767","Type":"ContainerStarted","Data":"9ac9c555d8fa938319184e80540a5ac6548c518ce1a7ec82569d7b853d131004"} Oct 31 21:58:44.037019 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.036991 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr"] Oct 31 21:58:44.045753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.045728 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec1ab11b-q2ccr"] Oct 31 21:58:44.797106 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.797079 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:44.864719 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.864680 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-kube-api-access-t9rqq\" (UniqueName: \"kubernetes.io/projected/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-kube-api-access-t9rqq\") pod \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " Oct 31 21:58:44.864919 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.864846 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-xks-ca\") pod \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " Oct 31 21:58:44.865050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.865027 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-openshift-ca\") pod \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\" (UID: \"827a8b47-7adf-4b07-be7c-fb4dc2bdb767\") " Oct 31 21:58:44.865540 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.865512 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-openshift-ca" pod "827a8b47-7adf-4b07-be7c-fb4dc2bdb767" (UID: "827a8b47-7adf-4b07-be7c-fb4dc2bdb767"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:44.867542 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.867511 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-kube-api-access-t9rqq" pod "827a8b47-7adf-4b07-be7c-fb4dc2bdb767" (UID: "827a8b47-7adf-4b07-be7c-fb4dc2bdb767"). InnerVolumeSpecName "kube-api-access-t9rqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:44.867652 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.867544 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-xks-ca" pod "827a8b47-7adf-4b07-be7c-fb4dc2bdb767" (UID: "827a8b47-7adf-4b07-be7c-fb4dc2bdb767"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:58:44.966778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.966741 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:44.966778 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.966778 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9rqq\" (UniqueName: \"kubernetes.io/projected/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-kube-api-access-t9rqq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:44.966981 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:44.966793 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/827a8b47-7adf-4b07-be7c-fb4dc2bdb767-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:45.632144 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:45.632111 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" Oct 31 21:58:45.632342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:45.632151 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf" event={"ID":"827a8b47-7adf-4b07-be7c-fb4dc2bdb767","Type":"ContainerDied","Data":"9ac9c555d8fa938319184e80540a5ac6548c518ce1a7ec82569d7b853d131004"} Oct 31 21:58:45.632342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:45.632212 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ac9c555d8fa938319184e80540a5ac6548c518ce1a7ec82569d7b853d131004" Oct 31 21:58:45.986194 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:45.986161 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400a2227-c7be-4132-b9db-609d9434df1f" path="/var/lib/kubelet/pods/400a2227-c7be-4132-b9db-609d9434df1f/volumes" Oct 31 21:58:46.031799 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:46.031768 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h"] Oct 31 21:58:46.035176 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:46.035149 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d85631e7-n2v2h"] Oct 31 21:58:46.969847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:46.969806 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl"] Oct 31 21:58:47.988914 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:47.988868 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4" path="/var/lib/kubelet/pods/5f7781c3-0e96-4dbc-ad4a-f3c9fb69d4b4/volumes" Oct 31 21:58:51.970862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:51.970824 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" podUID="fc1468cf-90bf-4368-9196-9346701a52d3" containerName="payload-processing" containerID="cri-o://9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532" gracePeriod=25 Oct 31 21:58:52.246199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.246171 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:52.313953 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.313924 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-routing-overlay-volume\") pod \"fc1468cf-90bf-4368-9196-9346701a52d3\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " Oct 31 21:58:52.314197 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.314177 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-kube-api-access-6nhsr\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-kube-api-access-6nhsr\") pod \"fc1468cf-90bf-4368-9196-9346701a52d3\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " Oct 31 21:58:52.314265 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.314230 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-plugins-config-volume\") pod \"fc1468cf-90bf-4368-9196-9346701a52d3\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " Oct 31 21:58:52.314387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.314360 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-routing-overlay-volume" pod "fc1468cf-90bf-4368-9196-9346701a52d3" (UID: "fc1468cf-90bf-4368-9196-9346701a52d3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:52.314387 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.314374 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-provider-credentials\") pod \"fc1468cf-90bf-4368-9196-9346701a52d3\" (UID: \"fc1468cf-90bf-4368-9196-9346701a52d3\") " Oct 31 21:58:52.314743 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.314711 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-plugins-config-volume" pod "fc1468cf-90bf-4368-9196-9346701a52d3" (UID: "fc1468cf-90bf-4368-9196-9346701a52d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:52.315108 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.315081 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:52.315208 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.315116 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fc1468cf-90bf-4368-9196-9346701a52d3-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:52.317198 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.317160 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-provider-credentials" pod "fc1468cf-90bf-4368-9196-9346701a52d3" (UID: "fc1468cf-90bf-4368-9196-9346701a52d3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:52.317331 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.317203 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-kube-api-access-6nhsr" pod "fc1468cf-90bf-4368-9196-9346701a52d3" (UID: "fc1468cf-90bf-4368-9196-9346701a52d3"). InnerVolumeSpecName "kube-api-access-6nhsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:52.416367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.416332 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nhsr\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-kube-api-access-6nhsr\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:52.416367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.416362 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fc1468cf-90bf-4368-9196-9346701a52d3-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:52.609368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.609335 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 21:58:52.687763 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.687733 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt"] Oct 31 21:58:52.708045 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.708005 4294 generic.go:356] "Generic (PLEG): container finished" podID="fc1468cf-90bf-4368-9196-9346701a52d3" containerID="9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532" exitCode=0 Oct 31 21:58:52.708283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.708125 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" event={"ID":"fc1468cf-90bf-4368-9196-9346701a52d3","Type":"ContainerDied","Data":"9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532"} Oct 31 21:58:52.708283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.708200 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" event={"ID":"fc1468cf-90bf-4368-9196-9346701a52d3","Type":"ContainerDied","Data":"5ec60a29e7196556b7ec1e80d0c3b5c503adf5f3b83f22b943b26d08cc10d9b4"} Oct 31 21:58:52.708283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.708231 4294 scope.go:122] "RemoveContainer" containerID="9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532" Oct 31 21:58:52.708572 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.708443 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl" Oct 31 21:58:52.721561 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.721537 4294 scope.go:122] "RemoveContainer" containerID="9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532" Oct 31 21:58:52.721928 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:58:52.721906 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532\": container with ID starting with 9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532 not found: ID does not exist" containerID="9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532" Oct 31 21:58:52.721993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.721939 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532"} err="failed to get container status \"9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532\": rpc error: code = NotFound desc = could not find container \"9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532\": container with ID starting with 9141ecb868f0e4866b0ae995ba2b5b997c23e98904ab7380cf56ad3d79b3c532 not found: ID does not exist" Oct 31 21:58:52.784727 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.784686 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl"] Oct 31 21:58:52.795253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:52.795023 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-86bblsjrl"] Oct 31 21:58:53.985788 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:53.985751 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1468cf-90bf-4368-9196-9346701a52d3" path="/var/lib/kubelet/pods/fc1468cf-90bf-4368-9196-9346701a52d3/volumes" Oct 31 21:58:54.032601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:54.032572 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt"] Oct 31 21:58:54.045296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:54.045271 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa47f95-hx9wt"] Oct 31 21:58:55.937548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.937513 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp"] Oct 31 21:58:55.938116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.938103 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fc1468cf-90bf-4368-9196-9346701a52d3" containerName="payload-processing" Oct 31 21:58:55.938172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.938117 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1468cf-90bf-4368-9196-9346701a52d3" containerName="payload-processing" Oct 31 21:58:55.938172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.938158 4294 cpu_manager.go:414] "containerMap: removing container" podUID="827a8b47-7adf-4b07-be7c-fb4dc2bdb767" containerName="revoke-keys" Oct 31 21:58:55.938172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.938163 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827a8b47-7adf-4b07-be7c-fb4dc2bdb767" containerName="revoke-keys" Oct 31 21:58:55.938283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.938273 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="827a8b47-7adf-4b07-be7c-fb4dc2bdb767" containerName="revoke-keys" Oct 31 21:58:55.938353 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.938288 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1468cf-90bf-4368-9196-9346701a52d3" containerName="payload-processing" Oct 31 21:58:55.943249 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.943231 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:55.959493 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.959468 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp"] Oct 31 21:58:55.985584 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:55.985553 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6de5d2a0-55f8-49a1-a095-c954e9937dde" path="/var/lib/kubelet/pods/6de5d2a0-55f8-49a1-a095-c954e9937dde/volumes" Oct 31 21:58:56.032865 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.032833 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp"] Oct 31 21:58:56.036085 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.036066 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5eeb6a0-xqtfp"] Oct 31 21:58:56.087296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.087266 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.087448 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.087423 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.087599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.087580 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.087644 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.087615 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xml8t\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-kube-api-access-xml8t\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.188747 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.188662 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.188921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.188769 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.188921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.188901 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.189111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.188927 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xml8t\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-kube-api-access-xml8t\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.189585 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.189559 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.189667 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.189569 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.191287 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.191264 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.213550 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.213530 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xml8t\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-kube-api-access-xml8t\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.257713 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.257690 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.559966 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.559938 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp"] Oct 31 21:58:56.560196 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:58:56.560174 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae9ae20_b2c9_4812_a03f_594bae477e89.slice/crio-8688f9f2a7e99fd20428e62f315255030c2135f2b133df64de06787122ed79c2 WatchSource:0}: Error finding container 8688f9f2a7e99fd20428e62f315255030c2135f2b133df64de06787122ed79c2: Status 404 returned error can't find the container with id 8688f9f2a7e99fd20428e62f315255030c2135f2b133df64de06787122ed79c2 Oct 31 21:58:56.748440 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.748399 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" event={"ID":"7ae9ae20-b2c9-4812-a03f-594bae477e89","Type":"ContainerStarted","Data":"8b65474e944dcd5364c0bdb23e7b8d4ac57b22207799adcc0e73b74b7076a935"} Oct 31 21:58:56.748611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.748482 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:58:56.748611 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.748502 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" event={"ID":"7ae9ae20-b2c9-4812-a03f-594bae477e89","Type":"ContainerStarted","Data":"8688f9f2a7e99fd20428e62f315255030c2135f2b133df64de06787122ed79c2"} Oct 31 21:58:56.798137 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:56.798086 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" podStartSLOduration=1.7980687450000001 podStartE2EDuration="1.798068745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:58:56.796745637 +0000 UTC m=+2576.921852877" watchObservedRunningTime="2026-10-31 21:58:56.798068745 +0000 UTC m=+2576.923175976" Oct 31 21:58:57.688711 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:57.688654 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" podUID="c2f7f141-2169-4d29-8c8a-0b272be18091" containerName="payload-processing" containerID="cri-o://1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4" gracePeriod=25 Oct 31 21:58:57.947762 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:57.947702 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:57.986917 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:57.986888 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4" path="/var/lib/kubelet/pods/b8fa9c0a-2a96-48d1-8a89-58b4c273d8d4/volumes" Oct 31 21:58:58.016079 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.016050 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-routing-overlay-volume\") pod \"c2f7f141-2169-4d29-8c8a-0b272be18091\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " Oct 31 21:58:58.016242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.016133 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-plugins-config-volume\") pod \"c2f7f141-2169-4d29-8c8a-0b272be18091\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " Oct 31 21:58:58.016242 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.016166 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-kube-api-access-76rsz\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-kube-api-access-76rsz\") pod \"c2f7f141-2169-4d29-8c8a-0b272be18091\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " Oct 31 21:58:58.016380 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.016261 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-provider-credentials\") pod \"c2f7f141-2169-4d29-8c8a-0b272be18091\" (UID: \"c2f7f141-2169-4d29-8c8a-0b272be18091\") " Oct 31 21:58:58.016720 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.016689 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-routing-overlay-volume" pod "c2f7f141-2169-4d29-8c8a-0b272be18091" (UID: "c2f7f141-2169-4d29-8c8a-0b272be18091"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:58.016720 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.016706 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-plugins-config-volume" pod "c2f7f141-2169-4d29-8c8a-0b272be18091" (UID: "c2f7f141-2169-4d29-8c8a-0b272be18091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:58:58.017051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.017009 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:58.017051 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.017037 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f7f141-2169-4d29-8c8a-0b272be18091-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:58.018738 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.018714 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-kube-api-access-76rsz" pod "c2f7f141-2169-4d29-8c8a-0b272be18091" (UID: "c2f7f141-2169-4d29-8c8a-0b272be18091"). InnerVolumeSpecName "kube-api-access-76rsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:58.018833 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.018764 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-provider-credentials" pod "c2f7f141-2169-4d29-8c8a-0b272be18091" (UID: "c2f7f141-2169-4d29-8c8a-0b272be18091"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:58:58.118083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.118045 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76rsz\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-kube-api-access-76rsz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:58.118083 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.118075 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c2f7f141-2169-4d29-8c8a-0b272be18091-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:58:58.763895 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.763860 4294 generic.go:356] "Generic (PLEG): container finished" podID="c2f7f141-2169-4d29-8c8a-0b272be18091" containerID="1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4" exitCode=0 Oct 31 21:58:58.764395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.763908 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" Oct 31 21:58:58.764395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.763946 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" event={"ID":"c2f7f141-2169-4d29-8c8a-0b272be18091","Type":"ContainerDied","Data":"1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4"} Oct 31 21:58:58.764395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.764005 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt" event={"ID":"c2f7f141-2169-4d29-8c8a-0b272be18091","Type":"ContainerDied","Data":"909cbb957192a7831d6e9fab7f3b499e7564ea7e2ce1f1ebdfa47bfdea0daae6"} Oct 31 21:58:58.764395 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.764034 4294 scope.go:122] "RemoveContainer" containerID="1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4" Oct 31 21:58:58.774601 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.774582 4294 scope.go:122] "RemoveContainer" containerID="1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4" Oct 31 21:58:58.774886 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:58:58.774858 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4\": container with ID starting with 1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4 not found: ID does not exist" containerID="1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4" Oct 31 21:58:58.774925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.774895 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4"} err="failed to get container status \"1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4\": rpc error: code = NotFound desc = could not find container \"1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4\": container with ID starting with 1c565f74f26f72558dd41efd78fedefa7353060bc91cd87050f83f9e641b19f4 not found: ID does not exist" Oct 31 21:58:58.821454 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.821428 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt"] Oct 31 21:58:58.833915 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:58.833891 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5ccchnpbt"] Oct 31 21:58:59.988773 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:58:59.988731 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f7f141-2169-4d29-8c8a-0b272be18091" path="/var/lib/kubelet/pods/c2f7f141-2169-4d29-8c8a-0b272be18091/volumes" Oct 31 21:59:01.310719 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:01.310682 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:59:01.822968 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.822930 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd"] Oct 31 21:59:01.823912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.823889 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c2f7f141-2169-4d29-8c8a-0b272be18091" containerName="payload-processing" Oct 31 21:59:01.823912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.823911 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f7f141-2169-4d29-8c8a-0b272be18091" containerName="payload-processing" Oct 31 21:59:01.824138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.824100 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f7f141-2169-4d29-8c8a-0b272be18091" containerName="payload-processing" Oct 31 21:59:01.829847 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.829821 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.850332 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.850282 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd"] Oct 31 21:59:01.858737 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.858711 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a5affe-openshift-default-dockercfg-f92tg\"" Oct 31 21:59:01.985988 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.985950 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istiod-ca-cert\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986074 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-data\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986133 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnl5s\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-kube-api-access-dnl5s\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986193 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986159 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-socket\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986254 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-podinfo\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986340 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-credential-socket\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986393 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986375 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-certs\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986565 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986543 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-envoy\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:01.986684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:01.986656 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-token\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097289 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istiod-ca-cert\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097430 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-data\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097464 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnl5s\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-kube-api-access-dnl5s\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097490 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-socket\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097522 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-podinfo\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097554 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-credential-socket\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097583 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-certs\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097722 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-envoy\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.097891 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-token\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.099417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.098875 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istiod-ca-cert\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.100050 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.099894 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-data\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.100466 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.100444 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-socket\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.104459 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.103935 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-credential-socket\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.104459 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.104176 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-certs\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.115560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.110672 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-podinfo\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.115560 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.115516 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-envoy\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.131735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.127599 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-token\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.131735 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.130552 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnl5s\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-kube-api-access-dnl5s\") pod \"e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.160649 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.160617 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:02.542770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.542739 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd"] Oct 31 21:59:02.543989 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:59:02.543960 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59a4817_e90a_4639_8abe_7f1584fa5eb6.slice/crio-ac4627d72b9d1e5a97a400e273b7d3197b9aef1f477dcddf6aecbb48b30a4c83 WatchSource:0}: Error finding container ac4627d72b9d1e5a97a400e273b7d3197b9aef1f477dcddf6aecbb48b30a4c83: Status 404 returned error can't find the container with id ac4627d72b9d1e5a97a400e273b7d3197b9aef1f477dcddf6aecbb48b30a4c83 Oct 31 21:59:02.547065 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.547032 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:59:02.547156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.547108 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:59:02.547156 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.547140 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 21:59:02.795548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.795431 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" event={"ID":"a59a4817-e90a-4639-8abe-7f1584fa5eb6","Type":"ContainerStarted","Data":"95d235eab51805239e7b995f5eeb1c7692cbc39ebdd6ba2089788d2411b31379"} Oct 31 21:59:02.795548 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.795488 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" event={"ID":"a59a4817-e90a-4639-8abe-7f1584fa5eb6","Type":"ContainerStarted","Data":"ac4627d72b9d1e5a97a400e273b7d3197b9aef1f477dcddf6aecbb48b30a4c83"} Oct 31 21:59:02.848052 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:02.847990 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" podStartSLOduration=1.8479702 podStartE2EDuration="1.8479702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:59:02.844073055 +0000 UTC m=+2582.969180286" watchObservedRunningTime="2026-10-31 21:59:02.8479702 +0000 UTC m=+2582.973077432" Oct 31 21:59:03.161286 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.161207 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:03.166868 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.166843 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:03.448464 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.448387 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5"] Oct 31 21:59:03.452700 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.452680 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.468858 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.468831 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:59:03.468993 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.468896 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:59:03.470739 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.470710 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5"] Oct 31 21:59:03.517663 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.517632 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.517829 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.517753 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.517875 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.517839 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rgnc\" (UniqueName: \"kubernetes.io/projected/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-kube-api-access-8rgnc\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.619368 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.619326 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rgnc\" (UniqueName: \"kubernetes.io/projected/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-kube-api-access-8rgnc\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.619770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.619540 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.619770 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.619612 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.620259 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.620232 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.621925 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.621892 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.643870 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.643844 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rgnc\" (UniqueName: \"kubernetes.io/projected/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-kube-api-access-8rgnc\") pod \"maas-api-revoke-sub-73b4f1eb-mbpf5\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.768537 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.768502 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:03.802671 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.802635 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:03.803823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:03.803800 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 21:59:04.072020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:04.071982 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5"] Oct 31 21:59:04.072401 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:59:04.072376 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d31e42_bc58_4c0f_b59a_3f48eb4608ad.slice/crio-b461cc85fe457bd710b076ac5f4ea4466ac5679922cf7fe46ea69c1369247387 WatchSource:0}: Error finding container b461cc85fe457bd710b076ac5f4ea4466ac5679922cf7fe46ea69c1369247387: Status 404 returned error can't find the container with id b461cc85fe457bd710b076ac5f4ea4466ac5679922cf7fe46ea69c1369247387 Oct 31 21:59:04.810285 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:04.810209 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" event={"ID":"42d31e42-bc58-4c0f-b59a-3f48eb4608ad","Type":"ContainerStarted","Data":"f33291453a48c51f597d7663370f7618162b41f4406f53c8d8771216286df051"} Oct 31 21:59:04.810772 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:04.810332 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" event={"ID":"42d31e42-bc58-4c0f-b59a-3f48eb4608ad","Type":"ContainerStarted","Data":"b461cc85fe457bd710b076ac5f4ea4466ac5679922cf7fe46ea69c1369247387"} Oct 31 21:59:04.856146 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:04.856098 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" podStartSLOduration=1.8560840170000001 podStartE2EDuration="1.856084017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:59:04.8523383 +0000 UTC m=+2584.977445531" watchObservedRunningTime="2026-10-31 21:59:04.856084017 +0000 UTC m=+2584.981191248" Oct 31 21:59:05.820199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:05.820160 4294 generic.go:356] "Generic (PLEG): container finished" podID="42d31e42-bc58-4c0f-b59a-3f48eb4608ad" containerID="f33291453a48c51f597d7663370f7618162b41f4406f53c8d8771216286df051" exitCode=0 Oct 31 21:59:05.820664 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:05.820457 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" event={"ID":"42d31e42-bc58-4c0f-b59a-3f48eb4608ad","Type":"ContainerDied","Data":"f33291453a48c51f597d7663370f7618162b41f4406f53c8d8771216286df051"} Oct 31 21:59:07.024933 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.024895 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:07.080506 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.080411 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-xks-ca\") pod \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " Oct 31 21:59:07.080749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.080607 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-kube-api-access-8rgnc\" (UniqueName: \"kubernetes.io/projected/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-kube-api-access-8rgnc\") pod \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " Oct 31 21:59:07.080749 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.080686 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-openshift-ca\") pod \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\" (UID: \"42d31e42-bc58-4c0f-b59a-3f48eb4608ad\") " Oct 31 21:59:07.081982 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.081955 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-openshift-ca" pod "42d31e42-bc58-4c0f-b59a-3f48eb4608ad" (UID: "42d31e42-bc58-4c0f-b59a-3f48eb4608ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:07.085086 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.085061 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-xks-ca" pod "42d31e42-bc58-4c0f-b59a-3f48eb4608ad" (UID: "42d31e42-bc58-4c0f-b59a-3f48eb4608ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:59:07.087294 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.087267 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-kube-api-access-8rgnc" pod "42d31e42-bc58-4c0f-b59a-3f48eb4608ad" (UID: "42d31e42-bc58-4c0f-b59a-3f48eb4608ad"). InnerVolumeSpecName "kube-api-access-8rgnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:07.182544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.182509 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:07.182544 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.182543 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rgnc\" (UniqueName: \"kubernetes.io/projected/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-kube-api-access-8rgnc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:07.182753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.182558 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42d31e42-bc58-4c0f-b59a-3f48eb4608ad-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:07.758642 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.758612 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:59:07.843507 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.843471 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj"] Oct 31 21:59:07.846643 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.846610 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" Oct 31 21:59:07.846806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.846720 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5" event={"ID":"42d31e42-bc58-4c0f-b59a-3f48eb4608ad","Type":"ContainerDied","Data":"b461cc85fe457bd710b076ac5f4ea4466ac5679922cf7fe46ea69c1369247387"} Oct 31 21:59:07.846806 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:07.846760 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b461cc85fe457bd710b076ac5f4ea4466ac5679922cf7fe46ea69c1369247387" Oct 31 21:59:10.859602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.859559 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx"] Oct 31 21:59:10.860555 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.860531 4294 cpu_manager.go:414] "containerMap: removing container" podUID="42d31e42-bc58-4c0f-b59a-3f48eb4608ad" containerName="revoke-keys" Oct 31 21:59:10.860662 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.860572 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42d31e42-bc58-4c0f-b59a-3f48eb4608ad" containerName="revoke-keys" Oct 31 21:59:10.860776 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.860763 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="42d31e42-bc58-4c0f-b59a-3f48eb4608ad" containerName="revoke-keys" Oct 31 21:59:10.864732 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.864708 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:10.882138 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.882106 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx"] Oct 31 21:59:10.892490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.889808 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4bd51b\"" Oct 31 21:59:10.892490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.889884 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"routing-overlay\"" Oct 31 21:59:10.892490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.889970 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"kube-root-ca.crt\"" Oct 31 21:59:10.892490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.890082 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"agc-cred-e47720795e77\"" Oct 31 21:59:10.892490 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.892151 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"payload-processing-external-model-e2e--c7d64253-dockercfg-lhtkj\"" Oct 31 21:59:10.892901 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.892873 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"openshift-service-ca.crt\"" Oct 31 21:59:10.962526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.962483 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:10.962709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.962628 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:10.962709 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.962666 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:10.962862 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:10.962795 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4nsx\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-kube-api-access-j4nsx\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.070289 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.070244 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.070602 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.070579 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.070824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.070807 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4nsx\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-kube-api-access-j4nsx\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.071098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.071078 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.076029 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.076002 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.096795 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.076812 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.101934 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.101906 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.121526 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.116984 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4nsx\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-kube-api-access-j4nsx\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.180182 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.180152 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.486660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.486633 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx"] Oct 31 21:59:11.488106 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:59:11.488070 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db679db_0408_47a2_8835_ace4f8a9ce17.slice/crio-b2e4f705b86dd7ade8c8b082790a9dd21cb1b47b21bd87e18a7c0469547f7170 WatchSource:0}: Error finding container b2e4f705b86dd7ade8c8b082790a9dd21cb1b47b21bd87e18a7c0469547f7170: Status 404 returned error can't find the container with id b2e4f705b86dd7ade8c8b082790a9dd21cb1b47b21bd87e18a7c0469547f7170 Oct 31 21:59:11.878823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.878721 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" event={"ID":"1db679db-0408-47a2-8835-ace4f8a9ce17","Type":"ContainerStarted","Data":"739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f"} Oct 31 21:59:11.878823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.878783 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" event={"ID":"1db679db-0408-47a2-8835-ace4f8a9ce17","Type":"ContainerStarted","Data":"b2e4f705b86dd7ade8c8b082790a9dd21cb1b47b21bd87e18a7c0469547f7170"} Oct 31 21:59:11.878823 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.878812 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:11.928324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:11.928180 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" podStartSLOduration=1.928158788 podStartE2EDuration="1.928158788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:59:11.925707685 +0000 UTC m=+2592.050814927" watchObservedRunningTime="2026-10-31 21:59:11.928158788 +0000 UTC m=+2592.053266020" Oct 31 21:59:11.994347 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:11.988171 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4bd51b/agc-cred-e47720795e77: secret "agc-cred-e47720795e77" not found Oct 31 21:59:11.994347 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:11.988199 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx: secret "agc-cred-e47720795e77" not found Oct 31 21:59:11.994347 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:11.988278 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials podName:1db679db-0408-47a2-8835-ace4f8a9ce17 nodeName:}" failed. No retries permitted until 2026-10-31 21:59:12.488255749 +0000 UTC m=+2592.613362977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17") : secret "agc-cred-e47720795e77" not found Oct 31 21:59:12.488828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.488794 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh"] Oct 31 21:59:12.493270 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.493247 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.510297 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.507866 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"e2e-cfl-a73b87-provider-cred\"" Oct 31 21:59:12.510297 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:12.508770 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4bd51b/agc-cred-e47720795e77: secret "agc-cred-e47720795e77" not found Oct 31 21:59:12.510297 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:12.508791 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx: secret "agc-cred-e47720795e77" not found Oct 31 21:59:12.510297 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:12.508863 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials podName:1db679db-0408-47a2-8835-ace4f8a9ce17 nodeName:}" failed. No retries permitted until 2026-10-31 21:59:13.508844944 +0000 UTC m=+2593.633952166 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17") : secret "agc-cred-e47720795e77" not found Oct 31 21:59:12.512781 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.512754 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh"] Oct 31 21:59:12.610427 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.610389 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.610619 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.610454 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.610684 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.610663 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.610731 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.610714 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2j7v\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-kube-api-access-w2j7v\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.712007 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.711972 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.712191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.712023 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.712191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.712154 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.712191 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.712189 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2j7v\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-kube-api-access-w2j7v\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.712849 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.712821 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.712975 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.712938 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.714524 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.714505 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.735265 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.735242 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2j7v\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-kube-api-access-w2j7v\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.809111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.809028 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:12.844434 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:12.844396 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" podUID="e0f68e3c-754d-4995-a177-759a637337bd" containerName="payload-processing" containerID="cri-o://e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8" gracePeriod=25 Oct 31 21:59:13.033485 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.033424 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l"] Oct 31 21:59:13.039060 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.039028 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720a0285-g2j7l"] Oct 31 21:59:13.111531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.111504 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh"] Oct 31 21:59:13.112283 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:59:13.112262 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c78436_f731_4d09_9d46_cfa4540c6b08.slice/crio-f94dd42face3d479174b9e0b16346fa4661f06482d7675651d054f6cf2a7fa27 WatchSource:0}: Error finding container f94dd42face3d479174b9e0b16346fa4661f06482d7675651d054f6cf2a7fa27: Status 404 returned error can't find the container with id f94dd42face3d479174b9e0b16346fa4661f06482d7675651d054f6cf2a7fa27 Oct 31 21:59:13.153946 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.153920 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:59:13.220580 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.220554 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-kube-api-access-5tfcc\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-kube-api-access-5tfcc\") pod \"e0f68e3c-754d-4995-a177-759a637337bd\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " Oct 31 21:59:13.220767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.220607 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-plugins-config-volume\") pod \"e0f68e3c-754d-4995-a177-759a637337bd\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " Oct 31 21:59:13.220767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.220757 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-provider-credentials\") pod \"e0f68e3c-754d-4995-a177-759a637337bd\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " Oct 31 21:59:13.220900 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.220815 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-routing-overlay-volume\") pod \"e0f68e3c-754d-4995-a177-759a637337bd\" (UID: \"e0f68e3c-754d-4995-a177-759a637337bd\") " Oct 31 21:59:13.221074 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.221052 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-plugins-config-volume" pod "e0f68e3c-754d-4995-a177-759a637337bd" (UID: "e0f68e3c-754d-4995-a177-759a637337bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:13.221417 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.221389 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-routing-overlay-volume" pod "e0f68e3c-754d-4995-a177-759a637337bd" (UID: "e0f68e3c-754d-4995-a177-759a637337bd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:13.221609 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.221587 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:13.221704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.221613 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0f68e3c-754d-4995-a177-759a637337bd-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:13.223067 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.223046 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-kube-api-access-5tfcc" pod "e0f68e3c-754d-4995-a177-759a637337bd" (UID: "e0f68e3c-754d-4995-a177-759a637337bd"). InnerVolumeSpecName "kube-api-access-5tfcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:13.223333 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.223295 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-provider-credentials" pod "e0f68e3c-754d-4995-a177-759a637337bd" (UID: "e0f68e3c-754d-4995-a177-759a637337bd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:13.322867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.322833 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:13.322867 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.322862 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tfcc\" (UniqueName: \"kubernetes.io/projected/e0f68e3c-754d-4995-a177-759a637337bd-kube-api-access-5tfcc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:13.526584 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:13.526554 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4bd51b/agc-cred-e47720795e77: secret "agc-cred-e47720795e77" not found Oct 31 21:59:13.526584 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:13.526580 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx: secret "agc-cred-e47720795e77" not found Oct 31 21:59:13.526825 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:13.526653 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials podName:1db679db-0408-47a2-8835-ace4f8a9ce17 nodeName:}" failed. No retries permitted until 2026-10-31 21:59:15.526634826 +0000 UTC m=+2595.651742052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17") : secret "agc-cred-e47720795e77" not found Oct 31 21:59:13.897334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.897216 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" event={"ID":"62c78436-f731-4d09-9d46-cfa4540c6b08","Type":"ContainerStarted","Data":"c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973"} Oct 31 21:59:13.897494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.897338 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" event={"ID":"62c78436-f731-4d09-9d46-cfa4540c6b08","Type":"ContainerStarted","Data":"f94dd42face3d479174b9e0b16346fa4661f06482d7675651d054f6cf2a7fa27"} Oct 31 21:59:13.897494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.897448 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:13.898694 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.898672 4294 generic.go:356] "Generic (PLEG): container finished" podID="e0f68e3c-754d-4995-a177-759a637337bd" containerID="e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8" exitCode=0 Oct 31 21:59:13.898813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.898695 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" event={"ID":"e0f68e3c-754d-4995-a177-759a637337bd","Type":"ContainerDied","Data":"e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8"} Oct 31 21:59:13.898813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.898732 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" Oct 31 21:59:13.898813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.898745 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj" event={"ID":"e0f68e3c-754d-4995-a177-759a637337bd","Type":"ContainerDied","Data":"8f26eb7a4955cbea8b97ac6f63b947331dd29956a0b8e72de40918aa9e895143"} Oct 31 21:59:13.898813 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.898776 4294 scope.go:122] "RemoveContainer" containerID="e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8" Oct 31 21:59:13.908987 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.908971 4294 scope.go:122] "RemoveContainer" containerID="e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8" Oct 31 21:59:13.909279 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:13.909259 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8\": container with ID starting with e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8 not found: ID does not exist" containerID="e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8" Oct 31 21:59:13.909367 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.909287 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8"} err="failed to get container status \"e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8\": rpc error: code = NotFound desc = could not find container \"e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8\": container with ID starting with e4b674a32933dfadb9d2932f21786b0f1490e39ea0aac20436ac1c371fbb62f8 not found: ID does not exist" Oct 31 21:59:13.939586 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.939545 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" podStartSLOduration=1.939532499 podStartE2EDuration="1.939532499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:59:13.938585754 +0000 UTC m=+2594.063692984" watchObservedRunningTime="2026-10-31 21:59:13.939532499 +0000 UTC m=+2594.064639730" Oct 31 21:59:13.976292 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.976270 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj"] Oct 31 21:59:13.985620 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.985598 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85479044-fab2-4413-a3f5-12d458699ec0" path="/var/lib/kubelet/pods/85479044-fab2-4413-a3f5-12d458699ec0/volumes" Oct 31 21:59:13.985912 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:13.985899 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6697482xj"] Oct 31 21:59:14.982766 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:14.982736 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m"] Oct 31 21:59:14.983378 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:14.983364 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e0f68e3c-754d-4995-a177-759a637337bd" containerName="payload-processing" Oct 31 21:59:14.983436 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:14.983379 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f68e3c-754d-4995-a177-759a637337bd" containerName="payload-processing" Oct 31 21:59:14.983489 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:14.983479 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f68e3c-754d-4995-a177-759a637337bd" containerName="payload-processing" Oct 31 21:59:14.988428 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:14.988411 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.004003 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.003982 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m"] Oct 31 21:59:15.012111 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.012085 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:59:15.012221 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.012161 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:59:15.047036 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.046996 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.047404 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.047380 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqqs2\" (UniqueName: \"kubernetes.io/projected/09868d24-c0a6-4979-af86-1b256e50eb3f-kube-api-access-xqqs2\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.047701 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.047682 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.149080 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.149049 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.149238 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.149161 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.149324 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.149292 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqqs2\" (UniqueName: \"kubernetes.io/projected/09868d24-c0a6-4979-af86-1b256e50eb3f-kube-api-access-xqqs2\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.149855 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.149831 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.151588 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.151566 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.173102 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.173073 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqqs2\" (UniqueName: \"kubernetes.io/projected/09868d24-c0a6-4979-af86-1b256e50eb3f-kube-api-access-xqqs2\") pod \"maas-api-revoke-sub-ace7a410-h9j4m\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.302004 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.301924 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:15.556246 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:15.556168 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4bd51b/agc-cred-e47720795e77: secret "agc-cred-e47720795e77" not found Oct 31 21:59:15.556246 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:15.556192 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx: secret "agc-cred-e47720795e77" not found Oct 31 21:59:15.556246 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:15.556246 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials podName:1db679db-0408-47a2-8835-ace4f8a9ce17 nodeName:}" failed. No retries permitted until 2026-10-31 21:59:19.556230906 +0000 UTC m=+2599.681338115 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17") : secret "agc-cred-e47720795e77" not found Oct 31 21:59:15.630199 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.630162 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m"] Oct 31 21:59:15.634541 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:59:15.634514 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09868d24_c0a6_4979_af86_1b256e50eb3f.slice/crio-25cd410d70a6a388b86921e3dc987e772f9a63674c8e28fefdccaab9aba99753 WatchSource:0}: Error finding container 25cd410d70a6a388b86921e3dc987e772f9a63674c8e28fefdccaab9aba99753: Status 404 returned error can't find the container with id 25cd410d70a6a388b86921e3dc987e772f9a63674c8e28fefdccaab9aba99753 Oct 31 21:59:15.916978 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.916946 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" event={"ID":"09868d24-c0a6-4979-af86-1b256e50eb3f","Type":"ContainerStarted","Data":"28f0fad6777f0a00f49f7390aaf305952c6bf67210201c11d468879d6c1f88e4"} Oct 31 21:59:15.917122 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.917001 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" event={"ID":"09868d24-c0a6-4979-af86-1b256e50eb3f","Type":"ContainerStarted","Data":"25cd410d70a6a388b86921e3dc987e772f9a63674c8e28fefdccaab9aba99753"} Oct 31 21:59:15.958819 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.958769 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" podStartSLOduration=1.958756314 podStartE2EDuration="1.958756314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 21:59:15.955738086 +0000 UTC m=+2596.080845316" watchObservedRunningTime="2026-10-31 21:59:15.958756314 +0000 UTC m=+2596.083863543" Oct 31 21:59:15.985334 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:15.985286 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f68e3c-754d-4995-a177-759a637337bd" path="/var/lib/kubelet/pods/e0f68e3c-754d-4995-a177-759a637337bd/volumes" Oct 31 21:59:16.925519 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:16.925489 4294 generic.go:356] "Generic (PLEG): container finished" podID="09868d24-c0a6-4979-af86-1b256e50eb3f" containerID="28f0fad6777f0a00f49f7390aaf305952c6bf67210201c11d468879d6c1f88e4" exitCode=0 Oct 31 21:59:16.925691 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:16.925577 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" event={"ID":"09868d24-c0a6-4979-af86-1b256e50eb3f","Type":"ContainerDied","Data":"28f0fad6777f0a00f49f7390aaf305952c6bf67210201c11d468879d6c1f88e4"} Oct 31 21:59:18.080439 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.080409 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:18.192832 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.192725 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09868d24-c0a6-4979-af86-1b256e50eb3f-kube-api-access-xqqs2\" (UniqueName: \"kubernetes.io/projected/09868d24-c0a6-4979-af86-1b256e50eb3f-kube-api-access-xqqs2\") pod \"09868d24-c0a6-4979-af86-1b256e50eb3f\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " Oct 31 21:59:18.193020 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.192963 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-xks-ca\") pod \"09868d24-c0a6-4979-af86-1b256e50eb3f\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " Oct 31 21:59:18.193136 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.193122 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-openshift-ca\") pod \"09868d24-c0a6-4979-af86-1b256e50eb3f\" (UID: \"09868d24-c0a6-4979-af86-1b256e50eb3f\") " Oct 31 21:59:18.193500 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.193471 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-openshift-ca" pod "09868d24-c0a6-4979-af86-1b256e50eb3f" (UID: "09868d24-c0a6-4979-af86-1b256e50eb3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:18.193641 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.193617 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:18.194950 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.194921 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09868d24-c0a6-4979-af86-1b256e50eb3f-kube-api-access-xqqs2" pod "09868d24-c0a6-4979-af86-1b256e50eb3f" (UID: "09868d24-c0a6-4979-af86-1b256e50eb3f"). InnerVolumeSpecName "kube-api-access-xqqs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:18.195075 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.195058 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-xks-ca" pod "09868d24-c0a6-4979-af86-1b256e50eb3f" (UID: "09868d24-c0a6-4979-af86-1b256e50eb3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:59:18.294562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.294528 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqqs2\" (UniqueName: \"kubernetes.io/projected/09868d24-c0a6-4979-af86-1b256e50eb3f-kube-api-access-xqqs2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:18.294562 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.294560 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09868d24-c0a6-4979-af86-1b256e50eb3f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:18.940834 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.940805 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" Oct 31 21:59:18.941027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.940837 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m" event={"ID":"09868d24-c0a6-4979-af86-1b256e50eb3f","Type":"ContainerDied","Data":"25cd410d70a6a388b86921e3dc987e772f9a63674c8e28fefdccaab9aba99753"} Oct 31 21:59:18.941027 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:18.940885 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25cd410d70a6a388b86921e3dc987e772f9a63674c8e28fefdccaab9aba99753" Oct 31 21:59:19.614664 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:19.614635 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4bd51b/agc-cred-e47720795e77: secret "agc-cred-e47720795e77" not found Oct 31 21:59:19.614664 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:19.614662 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx: secret "agc-cred-e47720795e77" not found Oct 31 21:59:19.615208 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:19.614729 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials podName:1db679db-0408-47a2-8835-ace4f8a9ce17 nodeName:}" failed. No retries permitted until 2026-10-31 21:59:27.614710231 +0000 UTC m=+2607.739817440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17") : secret "agc-cred-e47720795e77" not found Oct 31 21:59:22.892452 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:22.892418 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:23.033125 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:23.033095 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2"] Oct 31 21:59:23.036415 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:23.036389 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932716b2-qkjk2"] Oct 31 21:59:23.985642 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:23.985611 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3927850-0b21-4f67-a006-966a5239f32c" path="/var/lib/kubelet/pods/a3927850-0b21-4f67-a006-966a5239f32c/volumes" Oct 31 21:59:24.911941 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:24.911907 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 21:59:25.010118 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:25.010081 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx"] Oct 31 21:59:27.623408 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:27.623370 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-4bd51b/agc-cred-e47720795e77: secret "agc-cred-e47720795e77" not found Oct 31 21:59:27.623408 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:27.623400 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx: secret "agc-cred-e47720795e77" not found Oct 31 21:59:27.623945 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:27.623492 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials podName:1db679db-0408-47a2-8835-ace4f8a9ce17 nodeName:}" failed. No retries permitted until 2026-10-31 21:59:43.623471226 +0000 UTC m=+2623.748578449 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17") : secret "agc-cred-e47720795e77" not found Oct 31 21:59:28.563115 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.563074 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z"] Oct 31 21:59:28.563756 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.563742 4294 cpu_manager.go:414] "containerMap: removing container" podUID="09868d24-c0a6-4979-af86-1b256e50eb3f" containerName="revoke-keys" Oct 31 21:59:28.563810 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.563758 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09868d24-c0a6-4979-af86-1b256e50eb3f" containerName="revoke-keys" Oct 31 21:59:28.563883 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.563873 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="09868d24-c0a6-4979-af86-1b256e50eb3f" containerName="revoke-keys" Oct 31 21:59:28.567955 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.567939 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.585375 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.585350 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z"] Oct 31 21:59:28.592172 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.592147 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:59:28.592330 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.592198 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:59:28.644692 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.644658 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwrz\" (UniqueName: \"kubernetes.io/projected/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-kube-api-access-lmwrz\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.645070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.644870 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.645070 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.644989 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.746035 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.745997 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwrz\" (UniqueName: \"kubernetes.io/projected/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-kube-api-access-lmwrz\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.746205 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.746170 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.746275 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.746257 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.746972 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.746947 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.748712 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.748676 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.769632 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.769600 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwrz\" (UniqueName: \"kubernetes.io/projected/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-kube-api-access-lmwrz\") pod \"maas-api-revoke-sub-92d20848-nb26z\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:28.881264 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:28.881187 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:29.190444 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:29.190354 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z"] Oct 31 21:59:29.196743 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:59:29.196705 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3a010a_9bdf_4755_bdb1_aa19feddd7c8.slice/crio-6f7a70ba5eeff6665916a3a439edec12c867ba468925dd033a864284be03f646 WatchSource:0}: Error finding container 6f7a70ba5eeff6665916a3a439edec12c867ba468925dd033a864284be03f646: Status 404 returned error can't find the container with id 6f7a70ba5eeff6665916a3a439edec12c867ba468925dd033a864284be03f646 Oct 31 21:59:30.010511 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.010476 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" podUID="1db679db-0408-47a2-8835-ace4f8a9ce17" containerName="payload-processing" containerID="cri-o://739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f" gracePeriod=25 Oct 31 21:59:30.034523 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.034491 4294 generic.go:356] "Generic (PLEG): container finished" podID="fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" containerID="2339745ff972b92365fee11d89b56699c5268420b210ce1974e65b9dc35beacf" exitCode=0 Oct 31 21:59:30.034621 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.034573 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" event={"ID":"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8","Type":"ContainerDied","Data":"2339745ff972b92365fee11d89b56699c5268420b210ce1974e65b9dc35beacf"} Oct 31 21:59:30.034673 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.034635 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" event={"ID":"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8","Type":"ContainerStarted","Data":"6f7a70ba5eeff6665916a3a439edec12c867ba468925dd033a864284be03f646"} Oct 31 21:59:30.282008 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.281985 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:30.371503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.371166 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-plugins-config-volume\") pod \"1db679db-0408-47a2-8835-ace4f8a9ce17\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " Oct 31 21:59:30.371503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.371277 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-routing-overlay-volume\") pod \"1db679db-0408-47a2-8835-ace4f8a9ce17\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " Oct 31 21:59:30.371503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.371394 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials\") pod \"1db679db-0408-47a2-8835-ace4f8a9ce17\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " Oct 31 21:59:30.371503 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.371459 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-kube-api-access-j4nsx\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-kube-api-access-j4nsx\") pod \"1db679db-0408-47a2-8835-ace4f8a9ce17\" (UID: \"1db679db-0408-47a2-8835-ace4f8a9ce17\") " Oct 31 21:59:30.371857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.371630 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-routing-overlay-volume" pod "1db679db-0408-47a2-8835-ace4f8a9ce17" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:30.371857 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.371640 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-plugins-config-volume" pod "1db679db-0408-47a2-8835-ace4f8a9ce17" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:30.373599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.372152 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:30.373599 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.372175 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db679db-0408-47a2-8835-ace4f8a9ce17-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:30.374005 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.373983 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials" pod "1db679db-0408-47a2-8835-ace4f8a9ce17" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:30.374075 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.374057 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-kube-api-access-j4nsx" pod "1db679db-0408-47a2-8835-ace4f8a9ce17" (UID: "1db679db-0408-47a2-8835-ace4f8a9ce17"). InnerVolumeSpecName "kube-api-access-j4nsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:30.473660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.473629 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4nsx\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-kube-api-access-j4nsx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:30.473660 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:30.473657 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db679db-0408-47a2-8835-ace4f8a9ce17-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:31.042824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.042742 4294 generic.go:356] "Generic (PLEG): container finished" podID="1db679db-0408-47a2-8835-ace4f8a9ce17" containerID="739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f" exitCode=0 Oct 31 21:59:31.042824 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.042790 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" Oct 31 21:59:31.043491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.042825 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" event={"ID":"1db679db-0408-47a2-8835-ace4f8a9ce17","Type":"ContainerDied","Data":"739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f"} Oct 31 21:59:31.043491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.042889 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx" event={"ID":"1db679db-0408-47a2-8835-ace4f8a9ce17","Type":"ContainerDied","Data":"b2e4f705b86dd7ade8c8b082790a9dd21cb1b47b21bd87e18a7c0469547f7170"} Oct 31 21:59:31.043491 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.042919 4294 scope.go:122] "RemoveContainer" containerID="739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f" Oct 31 21:59:31.054169 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.054148 4294 scope.go:122] "RemoveContainer" containerID="739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f" Oct 31 21:59:31.054445 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:31.054428 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f\": container with ID starting with 739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f not found: ID does not exist" containerID="739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f" Oct 31 21:59:31.054512 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.054452 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f"} err="failed to get container status \"739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f\": rpc error: code = NotFound desc = could not find container \"739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f\": container with ID starting with 739505ac8071dd66c8d60c4c58aeb444ef73e61eb73c3f1c839e300b8a35199f not found: ID does not exist" Oct 31 21:59:31.095131 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.095104 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx"] Oct 31 21:59:31.108342 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.108315 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5689mttmx"] Oct 31 21:59:31.201828 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.201803 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:31.285149 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.285119 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-xks-ca\") pod \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " Oct 31 21:59:31.285340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.285219 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-kube-api-access-lmwrz\" (UniqueName: \"kubernetes.io/projected/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-kube-api-access-lmwrz\") pod \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " Oct 31 21:59:31.285340 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.285250 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-openshift-ca\") pod \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\" (UID: \"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8\") " Oct 31 21:59:31.285767 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.285742 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-openshift-ca" pod "fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" (UID: "fe3a010a-9bdf-4755-bdb1-aa19feddd7c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:31.285983 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.285962 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:31.287253 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.287223 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-xks-ca" pod "fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" (UID: "fe3a010a-9bdf-4755-bdb1-aa19feddd7c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 21:59:31.287704 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.287681 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-kube-api-access-lmwrz" pod "fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" (UID: "fe3a010a-9bdf-4755-bdb1-aa19feddd7c8"). InnerVolumeSpecName "kube-api-access-lmwrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:31.311477 ip-10-0-136-156 kubenswrapper[4294]: E1031 21:59:31.311450 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 21:59:31.387313 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.387279 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:31.387442 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.387318 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmwrz\" (UniqueName: \"kubernetes.io/projected/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8-kube-api-access-lmwrz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:31.985531 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:31.985498 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db679db-0408-47a2-8835-ace4f8a9ce17" path="/var/lib/kubelet/pods/1db679db-0408-47a2-8835-ace4f8a9ce17/volumes" Oct 31 21:59:32.051095 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:32.051069 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" Oct 31 21:59:32.051575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:32.051116 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z" event={"ID":"fe3a010a-9bdf-4755-bdb1-aa19feddd7c8","Type":"ContainerDied","Data":"6f7a70ba5eeff6665916a3a439edec12c867ba468925dd033a864284be03f646"} Oct 31 21:59:32.051575 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:32.051164 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f7a70ba5eeff6665916a3a439edec12c867ba468925dd033a864284be03f646" Oct 31 21:59:40.002088 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:40.002051 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp"] Oct 31 21:59:45.003296 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.003256 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" podUID="7ae9ae20-b2c9-4812-a03f-594bae477e89" containerName="payload-processing" containerID="cri-o://8b65474e944dcd5364c0bdb23e7b8d4ac57b22207799adcc0e73b74b7076a935" gracePeriod=25 Oct 31 21:59:45.146948 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.146702 4294 generic.go:356] "Generic (PLEG): container finished" podID="7ae9ae20-b2c9-4812-a03f-594bae477e89" containerID="8b65474e944dcd5364c0bdb23e7b8d4ac57b22207799adcc0e73b74b7076a935" exitCode=0 Oct 31 21:59:45.146948 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.146868 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" event={"ID":"7ae9ae20-b2c9-4812-a03f-594bae477e89","Type":"ContainerDied","Data":"8b65474e944dcd5364c0bdb23e7b8d4ac57b22207799adcc0e73b74b7076a935"} Oct 31 21:59:45.311384 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.311359 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:59:45.393098 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.393063 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-provider-credentials\") pod \"7ae9ae20-b2c9-4812-a03f-594bae477e89\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " Oct 31 21:59:45.393283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.393143 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-plugins-config-volume\") pod \"7ae9ae20-b2c9-4812-a03f-594bae477e89\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " Oct 31 21:59:45.393283 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.393253 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-kube-api-access-xml8t\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-kube-api-access-xml8t\") pod \"7ae9ae20-b2c9-4812-a03f-594bae477e89\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " Oct 31 21:59:45.393437 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.393415 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-routing-overlay-volume\") pod \"7ae9ae20-b2c9-4812-a03f-594bae477e89\" (UID: \"7ae9ae20-b2c9-4812-a03f-594bae477e89\") " Oct 31 21:59:45.393994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.393949 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-routing-overlay-volume" pod "7ae9ae20-b2c9-4812-a03f-594bae477e89" (UID: "7ae9ae20-b2c9-4812-a03f-594bae477e89"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:45.394116 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.394001 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-plugins-config-volume" pod "7ae9ae20-b2c9-4812-a03f-594bae477e89" (UID: "7ae9ae20-b2c9-4812-a03f-594bae477e89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 21:59:45.396260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.396121 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-provider-credentials" pod "7ae9ae20-b2c9-4812-a03f-594bae477e89" (UID: "7ae9ae20-b2c9-4812-a03f-594bae477e89"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:45.396260 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.396218 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-kube-api-access-xml8t" pod "7ae9ae20-b2c9-4812-a03f-594bae477e89" (UID: "7ae9ae20-b2c9-4812-a03f-594bae477e89"). InnerVolumeSpecName "kube-api-access-xml8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 21:59:45.496605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.496512 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:45.496605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.496549 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:45.496605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.496563 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae9ae20-b2c9-4812-a03f-594bae477e89-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:45.496605 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:45.496576 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xml8t\" (UniqueName: \"kubernetes.io/projected/7ae9ae20-b2c9-4812-a03f-594bae477e89-kube-api-access-xml8t\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 21:59:46.156994 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:46.156963 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" Oct 31 21:59:46.157494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:46.156991 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp" event={"ID":"7ae9ae20-b2c9-4812-a03f-594bae477e89","Type":"ContainerDied","Data":"8688f9f2a7e99fd20428e62f315255030c2135f2b133df64de06787122ed79c2"} Oct 31 21:59:46.157494 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:46.157050 4294 scope.go:122] "RemoveContainer" containerID="8b65474e944dcd5364c0bdb23e7b8d4ac57b22207799adcc0e73b74b7076a935" Oct 31 21:59:46.223889 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:46.223854 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp"] Oct 31 21:59:46.234554 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:46.234532 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5c97j2zpp"] Oct 31 21:59:47.986379 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:47.986344 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae9ae20-b2c9-4812-a03f-594bae477e89" path="/var/lib/kubelet/pods/7ae9ae20-b2c9-4812-a03f-594bae477e89/volumes" Oct 31 21:59:59.089921 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.089884 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s"] Oct 31 21:59:59.090790 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.090768 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" containerName="revoke-keys" Oct 31 21:59:59.090898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.090792 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" containerName="revoke-keys" Oct 31 21:59:59.090898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.090816 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7ae9ae20-b2c9-4812-a03f-594bae477e89" containerName="payload-processing" Oct 31 21:59:59.090898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.090824 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae9ae20-b2c9-4812-a03f-594bae477e89" containerName="payload-processing" Oct 31 21:59:59.090898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.090841 4294 cpu_manager.go:414] "containerMap: removing container" podUID="1db679db-0408-47a2-8835-ace4f8a9ce17" containerName="payload-processing" Oct 31 21:59:59.090898 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.090850 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db679db-0408-47a2-8835-ace4f8a9ce17" containerName="payload-processing" Oct 31 21:59:59.091163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.091032 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" containerName="revoke-keys" Oct 31 21:59:59.091163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.091047 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae9ae20-b2c9-4812-a03f-594bae477e89" containerName="payload-processing" Oct 31 21:59:59.091163 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.091060 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db679db-0408-47a2-8835-ace4f8a9ce17" containerName="payload-processing" Oct 31 21:59:59.094753 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.094733 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.111440 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.111408 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s"] Oct 31 21:59:59.119457 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.119431 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 21:59:59.119651 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.119508 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 21:59:59.164099 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.164065 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.164317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.164113 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.164317 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.164208 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5rj\" (UniqueName: \"kubernetes.io/projected/609de17a-3818-49a4-a0c6-3edc99e4337d-kube-api-access-js5rj\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.265713 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.265672 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.265932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.265725 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.265932 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.265786 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5rj\" (UniqueName: \"kubernetes.io/projected/609de17a-3818-49a4-a0c6-3edc99e4337d-kube-api-access-js5rj\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.266636 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.266614 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.268234 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.268208 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.293213 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.293183 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5rj\" (UniqueName: \"kubernetes.io/projected/609de17a-3818-49a4-a0c6-3edc99e4337d-kube-api-access-js5rj\") pod \"maas-api-revoke-sub-31b990fe-wsf8s\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.409114 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.409026 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 21:59:59.840311 ip-10-0-136-156 kubenswrapper[4294]: I1031 21:59:59.840268 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s"] Oct 31 21:59:59.841493 ip-10-0-136-156 kubenswrapper[4294]: W1031 21:59:59.841466 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609de17a_3818_49a4_a0c6_3edc99e4337d.slice/crio-6edfdd356fe15b086cf0581a12193c6339394cedf3ad7e2027d61973531832f2 WatchSource:0}: Error finding container 6edfdd356fe15b086cf0581a12193c6339394cedf3ad7e2027d61973531832f2: Status 404 returned error can't find the container with id 6edfdd356fe15b086cf0581a12193c6339394cedf3ad7e2027d61973531832f2 Oct 31 22:00:00.145090 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.145043 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b"] Oct 31 22:00:00.150240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.150210 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.150607 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.150581 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr"] Oct 31 22:00:00.155696 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.155673 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp"] Oct 31 22:00:00.155858 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.155796 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.159816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.159797 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.166490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.166461 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h"] Oct 31 22:00:00.172605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.172578 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv"] Oct 31 22:00:00.174758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.173129 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.182196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.182162 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b"] Oct 31 22:00:00.182555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.182271 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.186593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.186578 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr"] Oct 31 22:00:00.191118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.191081 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blgln\" (UniqueName: \"kubernetes.io/projected/95452683-4ed9-4672-9026-339d1a0a07e5-kube-api-access-blgln\") pod \"maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.191256 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.191147 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad066037-9e1a-48be-9db0-f3a9f8768a6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.191256 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.191186 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr4qv\" (UniqueName: \"kubernetes.io/projected/ad066037-9e1a-48be-9db0-f3a9f8768a6b-kube-api-access-rr4qv\") pod \"maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.191407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.191267 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b968b4-b2d0-4bc7-a8c1-d58735388295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.191548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.191509 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfzn\" (UniqueName: \"kubernetes.io/projected/e9b968b4-b2d0-4bc7-a8c1-d58735388295-kube-api-access-xvfzn\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.191626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.191554 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95452683-4ed9-4672-9026-339d1a0a07e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.192193 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.192077 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp"] Oct 31 22:00:00.197669 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.197031 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w"] Oct 31 22:00:00.210574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.210537 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h"] Oct 31 22:00:00.210807 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.210585 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp"] Oct 31 22:00:00.210908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.210887 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.216846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.215838 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv"] Oct 31 22:00:00.216846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.215940 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.221289 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.221266 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5"] Oct 31 22:00:00.233604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.233568 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.242442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.242412 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w"] Oct 31 22:00:00.246703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.246682 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4"] Oct 31 22:00:00.255834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.255810 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964"] Oct 31 22:00:00.273961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.271297 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.277133 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.277103 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2"] Oct 31 22:00:00.278320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.277391 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.287617 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.287592 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5"] Oct 31 22:00:00.287810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.287795 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp"] Oct 31 22:00:00.287932 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.287922 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb"] Oct 31 22:00:00.294416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.287815 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.296153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.295127 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr4qv\" (UniqueName: \"kubernetes.io/projected/ad066037-9e1a-48be-9db0-f3a9f8768a6b-kube-api-access-rr4qv\") pod \"maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.296153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.295181 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.296153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.295211 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decb8099-9e1b-4e62-8641-9769c2c9c83d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.296153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.295258 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8994495f-2a50-4a8b-af81-e5c14c57c4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.296153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.295288 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b968b4-b2d0-4bc7-a8c1-d58735388295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.296490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.296453 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b968b4-b2d0-4bc7-a8c1-d58735388295-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.296852 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.296825 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f793472-466a-4b0e-a83f-f21aa9d01c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.296985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.296869 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b742073-6765-4c13-931d-245b538180f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.296985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.296917 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfzn\" (UniqueName: \"kubernetes.io/projected/e9b968b4-b2d0-4bc7-a8c1-d58735388295-kube-api-access-xvfzn\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.297059 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.296981 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctk9z\" (UniqueName: \"kubernetes.io/projected/0b742073-6765-4c13-931d-245b538180f6-kube-api-access-ctk9z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.297059 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.297025 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95452683-4ed9-4672-9026-339d1a0a07e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.297119 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.297078 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a37b22-93fb-4781-a126-6a138e8ba5c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.297289 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.297270 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmqk2\" (UniqueName: \"kubernetes.io/projected/decb8099-9e1b-4e62-8641-9769c2c9c83d-kube-api-access-hmqk2\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.297627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.297608 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtd4s\" (UniqueName: \"kubernetes.io/projected/46a37b22-93fb-4781-a126-6a138e8ba5c7-kube-api-access-qtd4s\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.297854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.297837 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4mj\" (UniqueName: \"kubernetes.io/projected/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-kube-api-access-js4mj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.298013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.297993 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95452683-4ed9-4672-9026-339d1a0a07e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.298149 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.298126 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkf65\" (UniqueName: \"kubernetes.io/projected/8994495f-2a50-4a8b-af81-e5c14c57c4fc-kube-api-access-lkf65\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.298250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.298178 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blgln\" (UniqueName: \"kubernetes.io/projected/95452683-4ed9-4672-9026-339d1a0a07e5-kube-api-access-blgln\") pod \"maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.298250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.298223 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad066037-9e1a-48be-9db0-f3a9f8768a6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.298419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.298249 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6lv\" (UniqueName: \"kubernetes.io/projected/6f793472-466a-4b0e-a83f-f21aa9d01c92-kube-api-access-sh6lv\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.299116 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.299086 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad066037-9e1a-48be-9db0-f3a9f8768a6b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.299285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.299271 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4"] Oct 31 22:00:00.299403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.299323 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j"] Oct 31 22:00:00.299466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.299451 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.306343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.306264 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2"] Oct 31 22:00:00.306433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.306374 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.308550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.308527 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb"] Oct 31 22:00:00.311791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.311769 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j"] Oct 31 22:00:00.315031 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.315008 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964"] Oct 31 22:00:00.322939 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.322881 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfzn\" (UniqueName: \"kubernetes.io/projected/e9b968b4-b2d0-4bc7-a8c1-d58735388295-kube-api-access-xvfzn\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.322939 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.322898 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr4qv\" (UniqueName: \"kubernetes.io/projected/ad066037-9e1a-48be-9db0-f3a9f8768a6b-kube-api-access-rr4qv\") pod \"maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.323647 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.323629 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blgln\" (UniqueName: \"kubernetes.io/projected/95452683-4ed9-4672-9026-339d1a0a07e5-kube-api-access-blgln\") pod \"maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.399561 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399531 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz5hj\" (UniqueName: \"kubernetes.io/projected/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-kube-api-access-fz5hj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.399714 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399587 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzcsq\" (UniqueName: \"kubernetes.io/projected/86ef7678-8bf6-443c-90e1-8c1af8de24d7-kube-api-access-rzcsq\") pod \"maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.399781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399738 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2f6q\" (UniqueName: \"kubernetes.io/projected/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-kube-api-access-g2f6q\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.399838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399782 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2237055a-125a-45a4-998f-1ce799325f97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.399882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399862 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f793472-466a-4b0e-a83f-f21aa9d01c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.399931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399888 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b742073-6765-4c13-931d-245b538180f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.399968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399942 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctk9z\" (UniqueName: \"kubernetes.io/projected/0b742073-6765-4c13-931d-245b538180f6-kube-api-access-ctk9z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.400018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.399999 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ef7678-8bf6-443c-90e1-8c1af8de24d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.400057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400034 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a37b22-93fb-4781-a126-6a138e8ba5c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.400112 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400099 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmqk2\" (UniqueName: \"kubernetes.io/projected/decb8099-9e1b-4e62-8641-9769c2c9c83d-kube-api-access-hmqk2\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.400196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400174 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtd4s\" (UniqueName: \"kubernetes.io/projected/46a37b22-93fb-4781-a126-6a138e8ba5c7-kube-api-access-qtd4s\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.400258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400207 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4mj\" (UniqueName: \"kubernetes.io/projected/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-kube-api-access-js4mj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.400258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400238 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkf65\" (UniqueName: \"kubernetes.io/projected/8994495f-2a50-4a8b-af81-e5c14c57c4fc-kube-api-access-lkf65\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.400407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400263 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gch4b\" (UniqueName: \"kubernetes.io/projected/2237055a-125a-45a4-998f-1ce799325f97-kube-api-access-gch4b\") pod \"maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.400407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400293 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6lv\" (UniqueName: \"kubernetes.io/projected/6f793472-466a-4b0e-a83f-f21aa9d01c92-kube-api-access-sh6lv\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.400407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400331 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.400407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400373 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.400407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400390 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decb8099-9e1b-4e62-8641-9769c2c9c83d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.400712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400425 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8994495f-2a50-4a8b-af81-e5c14c57c4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.400712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400477 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.400712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400666 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f793472-466a-4b0e-a83f-f21aa9d01c92-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.400965 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.400939 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b742073-6765-4c13-931d-245b538180f6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.401158 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.401130 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" event={"ID":"609de17a-3818-49a4-a0c6-3edc99e4337d","Type":"ContainerStarted","Data":"9b81c58bc412de303391b266858128cb098b6f31736243c7aeda876d5b4b90f4"} Oct 31 22:00:00.401249 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.401187 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" event={"ID":"609de17a-3818-49a4-a0c6-3edc99e4337d","Type":"ContainerStarted","Data":"6edfdd356fe15b086cf0581a12193c6339394cedf3ad7e2027d61973531832f2"} Oct 31 22:00:00.401348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.401330 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8994495f-2a50-4a8b-af81-e5c14c57c4fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.401411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.401395 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.401641 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.401615 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decb8099-9e1b-4e62-8641-9769c2c9c83d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.402075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.402053 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a37b22-93fb-4781-a126-6a138e8ba5c7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.424568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.424537 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkf65\" (UniqueName: \"kubernetes.io/projected/8994495f-2a50-4a8b-af81-e5c14c57c4fc-kube-api-access-lkf65\") pod \"maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.424732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.424577 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtd4s\" (UniqueName: \"kubernetes.io/projected/46a37b22-93fb-4781-a126-6a138e8ba5c7-kube-api-access-qtd4s\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.424732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.424539 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmqk2\" (UniqueName: \"kubernetes.io/projected/decb8099-9e1b-4e62-8641-9769c2c9c83d-kube-api-access-hmqk2\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.424732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.424662 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4mj\" (UniqueName: \"kubernetes.io/projected/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-kube-api-access-js4mj\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.424732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.424683 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctk9z\" (UniqueName: \"kubernetes.io/projected/0b742073-6765-4c13-931d-245b538180f6-kube-api-access-ctk9z\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.425259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.425243 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6lv\" (UniqueName: \"kubernetes.io/projected/6f793472-466a-4b0e-a83f-f21aa9d01c92-kube-api-access-sh6lv\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.468287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.468257 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:00.481476 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.481452 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:00.501387 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.501353 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.501547 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.501402 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz5hj\" (UniqueName: \"kubernetes.io/projected/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-kube-api-access-fz5hj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.501547 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.501455 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcsq\" (UniqueName: \"kubernetes.io/projected/86ef7678-8bf6-443c-90e1-8c1af8de24d7-kube-api-access-rzcsq\") pod \"maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.501547 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.501531 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2f6q\" (UniqueName: \"kubernetes.io/projected/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-kube-api-access-g2f6q\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.501752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.501708 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2237055a-125a-45a4-998f-1ce799325f97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.501888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.501866 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ef7678-8bf6-443c-90e1-8c1af8de24d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.502895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.502025 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gch4b\" (UniqueName: \"kubernetes.io/projected/2237055a-125a-45a4-998f-1ce799325f97-kube-api-access-gch4b\") pod \"maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.502895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.502092 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.502895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.502102 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.502895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.502698 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ef7678-8bf6-443c-90e1-8c1af8de24d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.502895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.502853 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.503200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.503092 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2237055a-125a-45a4-998f-1ce799325f97-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.511504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.511483 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:00.525661 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.525634 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz5hj\" (UniqueName: \"kubernetes.io/projected/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-kube-api-access-fz5hj\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.526275 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.526246 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcsq\" (UniqueName: \"kubernetes.io/projected/86ef7678-8bf6-443c-90e1-8c1af8de24d7-kube-api-access-rzcsq\") pod \"maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.526432 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.526291 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2f6q\" (UniqueName: \"kubernetes.io/projected/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-kube-api-access-g2f6q\") pod \"maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.526432 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.526406 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gch4b\" (UniqueName: \"kubernetes.io/projected/2237055a-125a-45a4-998f-1ce799325f97-kube-api-access-gch4b\") pod \"maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.530108 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.530081 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:00.568918 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.568837 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:00.582343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.580762 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:00.594462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.594332 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:00.611631 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.611558 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:00.618708 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.618692 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:00.629876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.629811 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:00.683537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.683505 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:00.688555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.688527 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:00.698800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.698773 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:00.791182 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.791107 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" podStartSLOduration=1.791082244 podStartE2EDuration="1.791082244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 21:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:00.786454297 +0000 UTC m=+2640.911561529" watchObservedRunningTime="2026-10-31 22:00:00.791082244 +0000 UTC m=+2640.916189476" Oct 31 22:00:00.795318 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.795281 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b"] Oct 31 22:00:00.815557 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:00.808346 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad066037_9e1a_48be_9db0_f3a9f8768a6b.slice/crio-f9124f18011b80327b517b080ace591e9bafba03e45aaaf568bd97c36ab13881 WatchSource:0}: Error finding container f9124f18011b80327b517b080ace591e9bafba03e45aaaf568bd97c36ab13881: Status 404 returned error can't find the container with id f9124f18011b80327b517b080ace591e9bafba03e45aaaf568bd97c36ab13881 Oct 31 22:00:00.972181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:00.971681 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h"] Oct 31 22:00:00.993857 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:00.993811 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3be5fe_e715_4121_bd7b_e76b9fb6edbd.slice/crio-e439fa371a90d5ed838df6fc9ba89dcfa03b6fe5548501f0e8fa2176ac51b7d7 WatchSource:0}: Error finding container e439fa371a90d5ed838df6fc9ba89dcfa03b6fe5548501f0e8fa2176ac51b7d7: Status 404 returned error can't find the container with id e439fa371a90d5ed838df6fc9ba89dcfa03b6fe5548501f0e8fa2176ac51b7d7 Oct 31 22:00:01.023245 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:01.023211 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b968b4_b2d0_4bc7_a8c1_d58735388295.slice/crio-b20b25ce6d7431c48914d825ca240313c3b89176be73178dcc52b11d00425c4e WatchSource:0}: Error finding container b20b25ce6d7431c48914d825ca240313c3b89176be73178dcc52b11d00425c4e: Status 404 returned error can't find the container with id b20b25ce6d7431c48914d825ca240313c3b89176be73178dcc52b11d00425c4e Oct 31 22:00:01.030395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.030369 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr"] Oct 31 22:00:01.094911 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:01.094869 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95452683_4ed9_4672_9026_339d1a0a07e5.slice/crio-11e0bbc973ca075da7543a156802e3165efe9ea3bd964243ad00005dff77643b WatchSource:0}: Error finding container 11e0bbc973ca075da7543a156802e3165efe9ea3bd964243ad00005dff77643b: Status 404 returned error can't find the container with id 11e0bbc973ca075da7543a156802e3165efe9ea3bd964243ad00005dff77643b Oct 31 22:00:01.097057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.096429 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp"] Oct 31 22:00:01.315372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.315270 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5"] Oct 31 22:00:01.439053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.374543 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp"] Oct 31 22:00:01.439053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.379840 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w"] Oct 31 22:00:01.439053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.388925 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964"] Oct 31 22:00:01.557288 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:01.549119 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:00:01.569942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.568142 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4"] Oct 31 22:00:01.572758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.572728 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j"] Oct 31 22:00:01.586551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.586283 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv"] Oct 31 22:00:01.597683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.596523 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb"] Oct 31 22:00:01.607349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.607275 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2"] Oct 31 22:00:01.612614 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:01.612581 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b742073_6765_4c13_931d_245b538180f6.slice/crio-22196fc4896d654decb92097053b6eeb9075ecbb4b1e8d24f1c30551c4d1a396 WatchSource:0}: Error finding container 22196fc4896d654decb92097053b6eeb9075ecbb4b1e8d24f1c30551c4d1a396: Status 404 returned error can't find the container with id 22196fc4896d654decb92097053b6eeb9075ecbb4b1e8d24f1c30551c4d1a396 Oct 31 22:00:01.616153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.616117 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" event={"ID":"e9b968b4-b2d0-4bc7-a8c1-d58735388295","Type":"ContainerStarted","Data":"b20b25ce6d7431c48914d825ca240313c3b89176be73178dcc52b11d00425c4e"} Oct 31 22:00:01.626413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.626098 4294 generic.go:356] "Generic (PLEG): container finished" podID="609de17a-3818-49a4-a0c6-3edc99e4337d" containerID="9b81c58bc412de303391b266858128cb098b6f31736243c7aeda876d5b4b90f4" exitCode=0 Oct 31 22:00:01.626413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.626183 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" event={"ID":"609de17a-3818-49a4-a0c6-3edc99e4337d","Type":"ContainerDied","Data":"9b81c58bc412de303391b266858128cb098b6f31736243c7aeda876d5b4b90f4"} Oct 31 22:00:01.631579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.631440 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" event={"ID":"95452683-4ed9-4672-9026-339d1a0a07e5","Type":"ContainerStarted","Data":"11e0bbc973ca075da7543a156802e3165efe9ea3bd964243ad00005dff77643b"} Oct 31 22:00:01.638486 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.638454 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" event={"ID":"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd","Type":"ContainerStarted","Data":"e439fa371a90d5ed838df6fc9ba89dcfa03b6fe5548501f0e8fa2176ac51b7d7"} Oct 31 22:00:01.645538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.645435 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerStarted","Data":"06190620932f4d99ba549a3520790ce4f5964d58fcf56bfc46918252a35622b0"} Oct 31 22:00:01.645538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.645486 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerStarted","Data":"f9124f18011b80327b517b080ace591e9bafba03e45aaaf568bd97c36ab13881"} Oct 31 22:00:01.647816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.647791 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerStarted","Data":"fac0724f9c3c33bd5f422229712eb33038a125f235120ecccbcefd889bab7444"} Oct 31 22:00:01.710208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.710157 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" podStartSLOduration=1.7101375079999999 podStartE2EDuration="1.710137508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:01.708561861 +0000 UTC m=+2641.833669093" watchObservedRunningTime="2026-10-31 22:00:01.710137508 +0000 UTC m=+2641.835244740" Oct 31 22:00:01.752315 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:01.751998 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" podStartSLOduration=1.7519808540000001 podStartE2EDuration="1.751980854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:01.751596638 +0000 UTC m=+2641.876703868" watchObservedRunningTime="2026-10-31 22:00:01.751980854 +0000 UTC m=+2641.877088086" Oct 31 22:00:02.221224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.221198 4294 scope.go:122] "RemoveContainer" containerID="91b1e77300a531f7d3c9e59b42f085638396282a0540841c08a73850e66a4626" Oct 31 22:00:02.385883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.385853 4294 scope.go:122] "RemoveContainer" containerID="d8d96d68da340cec2e0d6a455f73614ab093bce47ab733ec7774ee3f3211dda6" Oct 31 22:00:02.395292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.395266 4294 scope.go:122] "RemoveContainer" containerID="1534c5ec4e8731d6feec82481416fc111d622dca2308a955e5f9322bdb20c033" Oct 31 22:00:02.404297 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.404280 4294 scope.go:122] "RemoveContainer" containerID="0d7f4ba4351e6f5b4868d9b88e4c0c627eccc21775aba89eb00ce1d5df9895a9" Oct 31 22:00:02.657440 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.657285 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerStarted","Data":"406a7f39ccbef3bd3a82ef99d1eb47cf09791c01fe4abd299bc6b0bcdf26fad0"} Oct 31 22:00:02.659025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.658998 4294 generic.go:356] "Generic (PLEG): container finished" podID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerID="154e84ff215a075d5b8807963edf0168fa5b43d82bd0f56b2018ba4740c5b208" exitCode=60 Oct 31 22:00:02.659182 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.659075 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" event={"ID":"e9b968b4-b2d0-4bc7-a8c1-d58735388295","Type":"ContainerDied","Data":"154e84ff215a075d5b8807963edf0168fa5b43d82bd0f56b2018ba4740c5b208"} Oct 31 22:00:02.659382 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.659362 4294 scope.go:122] "RemoveContainer" containerID="154e84ff215a075d5b8807963edf0168fa5b43d82bd0f56b2018ba4740c5b208" Oct 31 22:00:02.661434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.661412 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerStarted","Data":"9a99d72ce517d991bb12d1f29d3e82a874a5b4adf70bb3b1145a30ed4917456f"} Oct 31 22:00:02.661537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.661459 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerStarted","Data":"2249a7958410caa9c9ba276edca54f7dccef31cd1a723d4f9ff74b932a5f4a09"} Oct 31 22:00:02.667028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.667001 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerStarted","Data":"c38e581f0906df474f814eb515386044100f4a7299b13fd947f49636e17999b2"} Oct 31 22:00:02.667143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.667048 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerStarted","Data":"64e07d927082cb82fdd7adb5b88453bc64c5334cf4f63438b0916fca35e85323"} Oct 31 22:00:02.668687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.668661 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerStarted","Data":"bec3c6ffa5765bc8c1102ecb2b285000780f9b7a65fca1829959ab80bda8a55a"} Oct 31 22:00:02.668791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.668709 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerStarted","Data":"2cfde700f8f2688117b73684c6880a8c6a0a7ea9a17d1f6d04d8b5d69fe592bb"} Oct 31 22:00:02.670049 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.670024 4294 generic.go:356] "Generic (PLEG): container finished" podID="95452683-4ed9-4672-9026-339d1a0a07e5" containerID="ac71c65aee79a12e83bc3df53645292c23afc79dc312df96a09a2992d3630ccd" exitCode=60 Oct 31 22:00:02.670133 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.670093 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" event={"ID":"95452683-4ed9-4672-9026-339d1a0a07e5","Type":"ContainerDied","Data":"ac71c65aee79a12e83bc3df53645292c23afc79dc312df96a09a2992d3630ccd"} Oct 31 22:00:02.670266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.670254 4294 scope.go:122] "RemoveContainer" containerID="ac71c65aee79a12e83bc3df53645292c23afc79dc312df96a09a2992d3630ccd" Oct 31 22:00:02.673266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.672698 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerStarted","Data":"509ec8b4c31b35fc7cda1ff6dfc8714d03aa67e7da2bbfcd13bcc82aca4cc5ac"} Oct 31 22:00:02.673266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.672742 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerStarted","Data":"9b4154db53dffbdc223d1179ff53585c27982362cb73f32508417e50a1c4c61b"} Oct 31 22:00:02.683022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.682941 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" event={"ID":"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713","Type":"ContainerStarted","Data":"7d549a95cf746c4e03851d6a163e87a81aa9ac1278e861fd761ccf4c6a294f20"} Oct 31 22:00:02.683022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.682992 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" event={"ID":"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713","Type":"ContainerStarted","Data":"d4ae3e1643bda5bb491ab23cf25ee9a103c32464e85c04fc15f6d97643ca50f5"} Oct 31 22:00:02.685157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.685133 4294 generic.go:356] "Generic (PLEG): container finished" podID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerID="b6bbbcf4ab47c551099b6d173e54e2dfcc1c4c95ad7f2f2bbf93df6cc2588312" exitCode=60 Oct 31 22:00:02.685246 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.685202 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" event={"ID":"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd","Type":"ContainerDied","Data":"b6bbbcf4ab47c551099b6d173e54e2dfcc1c4c95ad7f2f2bbf93df6cc2588312"} Oct 31 22:00:02.685435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.685422 4294 scope.go:122] "RemoveContainer" containerID="b6bbbcf4ab47c551099b6d173e54e2dfcc1c4c95ad7f2f2bbf93df6cc2588312" Oct 31 22:00:02.686974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.686954 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerStarted","Data":"8900cc7705aa47e712412d57bdbebc944b5b31c045c8839fc10957abcfcc08ec"} Oct 31 22:00:02.687076 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.686987 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerStarted","Data":"383a65ab8cf3272850c8d97251986c6b1f9959339a3e2ecb1149191768b1c453"} Oct 31 22:00:02.688636 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.688616 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerStarted","Data":"f89c601f883c0177072e76ea28ed55df0842281e81f69858b4be4b41b51d8192"} Oct 31 22:00:02.688734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.688653 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerStarted","Data":"22196fc4896d654decb92097053b6eeb9075ecbb4b1e8d24f1c30551c4d1a396"} Oct 31 22:00:02.690176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.690155 4294 generic.go:356] "Generic (PLEG): container finished" podID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerID="06190620932f4d99ba549a3520790ce4f5964d58fcf56bfc46918252a35622b0" exitCode=60 Oct 31 22:00:02.690351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.690232 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerDied","Data":"06190620932f4d99ba549a3520790ce4f5964d58fcf56bfc46918252a35622b0"} Oct 31 22:00:02.690471 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.690454 4294 scope.go:122] "RemoveContainer" containerID="06190620932f4d99ba549a3520790ce4f5964d58fcf56bfc46918252a35622b0" Oct 31 22:00:02.691930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.691907 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerStarted","Data":"712d50fe9ce69dc3c221cb869f8cf4bc1057f00e0800b109da5fcca493532ef6"} Oct 31 22:00:02.692028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.691945 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerStarted","Data":"2f32f680a70977ab02faabe862e32c85d24637545c5655fe2f90e0787f08728e"} Oct 31 22:00:02.701618 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.701577 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" podStartSLOduration=2.70156469 podStartE2EDuration="2.70156469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:02.696823902 +0000 UTC m=+2642.821931132" watchObservedRunningTime="2026-10-31 22:00:02.70156469 +0000 UTC m=+2642.826671920" Oct 31 22:00:02.740397 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.740349 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" podStartSLOduration=2.74033356 podStartE2EDuration="2.74033356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:02.738495898 +0000 UTC m=+2642.863603131" watchObservedRunningTime="2026-10-31 22:00:02.74033356 +0000 UTC m=+2642.865440785" Oct 31 22:00:02.787969 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.783207 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" podStartSLOduration=2.783187244 podStartE2EDuration="2.783187244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:02.780726969 +0000 UTC m=+2642.905834200" watchObservedRunningTime="2026-10-31 22:00:02.783187244 +0000 UTC m=+2642.908294475" Oct 31 22:00:02.908184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.908079 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" podStartSLOduration=2.907360663 podStartE2EDuration="2.907360663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:02.900932775 +0000 UTC m=+2643.026040034" watchObservedRunningTime="2026-10-31 22:00:02.907360663 +0000 UTC m=+2643.032467896" Oct 31 22:00:02.947226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.947183 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" podStartSLOduration=2.947169719 podStartE2EDuration="2.947169719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:02.944105963 +0000 UTC m=+2643.069213193" watchObservedRunningTime="2026-10-31 22:00:02.947169719 +0000 UTC m=+2643.072276943" Oct 31 22:00:02.987105 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:02.987059 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" podStartSLOduration=2.987043993 podStartE2EDuration="2.987043993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:02.983911391 +0000 UTC m=+2643.109018621" watchObservedRunningTime="2026-10-31 22:00:02.987043993 +0000 UTC m=+2643.112151223" Oct 31 22:00:03.070147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.070082 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" podStartSLOduration=3.0700634940000002 podStartE2EDuration="3.070063494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:03.062569839 +0000 UTC m=+2643.187677069" watchObservedRunningTime="2026-10-31 22:00:03.070063494 +0000 UTC m=+2643.195170728" Oct 31 22:00:03.103946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.103903 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" podStartSLOduration=3.103889684 podStartE2EDuration="3.103889684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:03.102065257 +0000 UTC m=+2643.227172499" watchObservedRunningTime="2026-10-31 22:00:03.103889684 +0000 UTC m=+2643.228996914" Oct 31 22:00:03.174223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.174157 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 22:00:03.207480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.207454 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-xks-ca\") pod \"609de17a-3818-49a4-a0c6-3edc99e4337d\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " Oct 31 22:00:03.207608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.207590 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/609de17a-3818-49a4-a0c6-3edc99e4337d-kube-api-access-js5rj\" (UniqueName: \"kubernetes.io/projected/609de17a-3818-49a4-a0c6-3edc99e4337d-kube-api-access-js5rj\") pod \"609de17a-3818-49a4-a0c6-3edc99e4337d\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " Oct 31 22:00:03.207671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.207619 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-openshift-ca\") pod \"609de17a-3818-49a4-a0c6-3edc99e4337d\" (UID: \"609de17a-3818-49a4-a0c6-3edc99e4337d\") " Oct 31 22:00:03.208058 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.208032 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-openshift-ca" pod "609de17a-3818-49a4-a0c6-3edc99e4337d" (UID: "609de17a-3818-49a4-a0c6-3edc99e4337d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:03.208233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.208218 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:03.209494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.209465 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-xks-ca" pod "609de17a-3818-49a4-a0c6-3edc99e4337d" (UID: "609de17a-3818-49a4-a0c6-3edc99e4337d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:00:03.209598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.209579 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609de17a-3818-49a4-a0c6-3edc99e4337d-kube-api-access-js5rj" pod "609de17a-3818-49a4-a0c6-3edc99e4337d" (UID: "609de17a-3818-49a4-a0c6-3edc99e4337d"). InnerVolumeSpecName "kube-api-access-js5rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:03.225763 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.225719 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" podStartSLOduration=3.225705609 podStartE2EDuration="3.225705609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:03.185685778 +0000 UTC m=+2643.310793020" watchObservedRunningTime="2026-10-31 22:00:03.225705609 +0000 UTC m=+2643.350812851" Oct 31 22:00:03.309974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.309932 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/609de17a-3818-49a4-a0c6-3edc99e4337d-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:03.309974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.309966 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js5rj\" (UniqueName: \"kubernetes.io/projected/609de17a-3818-49a4-a0c6-3edc99e4337d-kube-api-access-js5rj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:03.701272 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.701241 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" event={"ID":"95452683-4ed9-4672-9026-339d1a0a07e5","Type":"ContainerStarted","Data":"e945d36dfca1e986c98ce484d50d4b095235dd5f186a008904b6c9c00b8f79b0"} Oct 31 22:00:03.702868 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.702846 4294 generic.go:356] "Generic (PLEG): container finished" podID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerID="509ec8b4c31b35fc7cda1ff6dfc8714d03aa67e7da2bbfcd13bcc82aca4cc5ac" exitCode=60 Oct 31 22:00:03.702972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.702923 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerDied","Data":"509ec8b4c31b35fc7cda1ff6dfc8714d03aa67e7da2bbfcd13bcc82aca4cc5ac"} Oct 31 22:00:03.703158 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.703143 4294 scope.go:122] "RemoveContainer" containerID="509ec8b4c31b35fc7cda1ff6dfc8714d03aa67e7da2bbfcd13bcc82aca4cc5ac" Oct 31 22:00:03.704554 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.704535 4294 generic.go:356] "Generic (PLEG): container finished" podID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerID="7d549a95cf746c4e03851d6a163e87a81aa9ac1278e861fd761ccf4c6a294f20" exitCode=60 Oct 31 22:00:03.704660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.704617 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" event={"ID":"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713","Type":"ContainerDied","Data":"7d549a95cf746c4e03851d6a163e87a81aa9ac1278e861fd761ccf4c6a294f20"} Oct 31 22:00:03.704863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.704811 4294 scope.go:122] "RemoveContainer" containerID="7d549a95cf746c4e03851d6a163e87a81aa9ac1278e861fd761ccf4c6a294f20" Oct 31 22:00:03.706786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.706765 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" event={"ID":"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd","Type":"ContainerStarted","Data":"d72f659f125bcea9a363dc623881bcf7c45f0ae25116407f36f805d79393d79e"} Oct 31 22:00:03.708416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.708398 4294 generic.go:356] "Generic (PLEG): container finished" podID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerID="8900cc7705aa47e712412d57bdbebc944b5b31c045c8839fc10957abcfcc08ec" exitCode=60 Oct 31 22:00:03.708505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.708479 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerDied","Data":"8900cc7705aa47e712412d57bdbebc944b5b31c045c8839fc10957abcfcc08ec"} Oct 31 22:00:03.708681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.708660 4294 scope.go:122] "RemoveContainer" containerID="8900cc7705aa47e712412d57bdbebc944b5b31c045c8839fc10957abcfcc08ec" Oct 31 22:00:03.710069 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.710050 4294 generic.go:356] "Generic (PLEG): container finished" podID="0b742073-6765-4c13-931d-245b538180f6" containerID="f89c601f883c0177072e76ea28ed55df0842281e81f69858b4be4b41b51d8192" exitCode=60 Oct 31 22:00:03.710188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.710128 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerDied","Data":"f89c601f883c0177072e76ea28ed55df0842281e81f69858b4be4b41b51d8192"} Oct 31 22:00:03.710354 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.710323 4294 scope.go:122] "RemoveContainer" containerID="f89c601f883c0177072e76ea28ed55df0842281e81f69858b4be4b41b51d8192" Oct 31 22:00:03.712121 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.712100 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerStarted","Data":"ba8a4fa6b882d63b72436edee5bea72f1d4cff677932c7d5362524985ccb3d47"} Oct 31 22:00:03.713565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.713543 4294 generic.go:356] "Generic (PLEG): container finished" podID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerID="712d50fe9ce69dc3c221cb869f8cf4bc1057f00e0800b109da5fcca493532ef6" exitCode=60 Oct 31 22:00:03.713653 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.713618 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerDied","Data":"712d50fe9ce69dc3c221cb869f8cf4bc1057f00e0800b109da5fcca493532ef6"} Oct 31 22:00:03.714290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.713804 4294 scope.go:122] "RemoveContainer" containerID="712d50fe9ce69dc3c221cb869f8cf4bc1057f00e0800b109da5fcca493532ef6" Oct 31 22:00:03.715677 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.715653 4294 generic.go:356] "Generic (PLEG): container finished" podID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerID="406a7f39ccbef3bd3a82ef99d1eb47cf09791c01fe4abd299bc6b0bcdf26fad0" exitCode=60 Oct 31 22:00:03.715780 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.715737 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerDied","Data":"406a7f39ccbef3bd3a82ef99d1eb47cf09791c01fe4abd299bc6b0bcdf26fad0"} Oct 31 22:00:03.715902 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.715884 4294 scope.go:122] "RemoveContainer" containerID="406a7f39ccbef3bd3a82ef99d1eb47cf09791c01fe4abd299bc6b0bcdf26fad0" Oct 31 22:00:03.717866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.717843 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" event={"ID":"e9b968b4-b2d0-4bc7-a8c1-d58735388295","Type":"ContainerStarted","Data":"a7475924cf0353cd252a9d167476ff533f798c4c030c225369a88ecc04f5105b"} Oct 31 22:00:03.719559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.719529 4294 generic.go:356] "Generic (PLEG): container finished" podID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerID="9a99d72ce517d991bb12d1f29d3e82a874a5b4adf70bb3b1145a30ed4917456f" exitCode=60 Oct 31 22:00:03.719671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.719606 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerDied","Data":"9a99d72ce517d991bb12d1f29d3e82a874a5b4adf70bb3b1145a30ed4917456f"} Oct 31 22:00:03.719778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.719754 4294 scope.go:122] "RemoveContainer" containerID="9a99d72ce517d991bb12d1f29d3e82a874a5b4adf70bb3b1145a30ed4917456f" Oct 31 22:00:03.720955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.720936 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" Oct 31 22:00:03.721045 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.720935 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s" event={"ID":"609de17a-3818-49a4-a0c6-3edc99e4337d","Type":"ContainerDied","Data":"6edfdd356fe15b086cf0581a12193c6339394cedf3ad7e2027d61973531832f2"} Oct 31 22:00:03.721096 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.721059 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6edfdd356fe15b086cf0581a12193c6339394cedf3ad7e2027d61973531832f2" Oct 31 22:00:03.724870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.724537 4294 generic.go:356] "Generic (PLEG): container finished" podID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerID="c38e581f0906df474f814eb515386044100f4a7299b13fd947f49636e17999b2" exitCode=60 Oct 31 22:00:03.724870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.724617 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerDied","Data":"c38e581f0906df474f814eb515386044100f4a7299b13fd947f49636e17999b2"} Oct 31 22:00:03.725027 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.724913 4294 scope.go:122] "RemoveContainer" containerID="c38e581f0906df474f814eb515386044100f4a7299b13fd947f49636e17999b2" Oct 31 22:00:03.728113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.728097 4294 generic.go:356] "Generic (PLEG): container finished" podID="2237055a-125a-45a4-998f-1ce799325f97" containerID="bec3c6ffa5765bc8c1102ecb2b285000780f9b7a65fca1829959ab80bda8a55a" exitCode=60 Oct 31 22:00:03.728213 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.728125 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerDied","Data":"bec3c6ffa5765bc8c1102ecb2b285000780f9b7a65fca1829959ab80bda8a55a"} Oct 31 22:00:03.728451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.728434 4294 scope.go:122] "RemoveContainer" containerID="bec3c6ffa5765bc8c1102ecb2b285000780f9b7a65fca1829959ab80bda8a55a" Oct 31 22:00:03.744070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:03.744024 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" podStartSLOduration=3.744007318 podStartE2EDuration="3.744007318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:03.740864015 +0000 UTC m=+2643.865971247" watchObservedRunningTime="2026-10-31 22:00:03.744007318 +0000 UTC m=+2643.869114550" Oct 31 22:00:04.248534 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.248449 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" podStartSLOduration=4.248435786 podStartE2EDuration="4.248435786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:04.244694141 +0000 UTC m=+2644.369801374" watchObservedRunningTime="2026-10-31 22:00:04.248435786 +0000 UTC m=+2644.373543016" Oct 31 22:00:04.742196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.742166 4294 generic.go:356] "Generic (PLEG): container finished" podID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerID="d72f659f125bcea9a363dc623881bcf7c45f0ae25116407f36f805d79393d79e" exitCode=60 Oct 31 22:00:04.742810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.742254 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" event={"ID":"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd","Type":"ContainerDied","Data":"d72f659f125bcea9a363dc623881bcf7c45f0ae25116407f36f805d79393d79e"} Oct 31 22:00:04.742810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.742394 4294 scope.go:122] "RemoveContainer" containerID="b6bbbcf4ab47c551099b6d173e54e2dfcc1c4c95ad7f2f2bbf93df6cc2588312" Oct 31 22:00:04.742810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.742609 4294 scope.go:122] "RemoveContainer" containerID="d72f659f125bcea9a363dc623881bcf7c45f0ae25116407f36f805d79393d79e" Oct 31 22:00:04.743073 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:04.743037 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h_odh-ai-gateway-infra(ec3be5fe-e715-4121-bd7b-e76b9fb6edbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" Oct 31 22:00:04.744666 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.744640 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerStarted","Data":"26484b008efef826e1f9f89c1e0c101f9c86107824e8cbf99f31f389e17e4b01"} Oct 31 22:00:04.746995 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.746975 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerStarted","Data":"5109bb6c8e15394b01d8d3c81d51390758f60166b611881c5927e1feb2e41486"} Oct 31 22:00:04.749733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.749713 4294 generic.go:356] "Generic (PLEG): container finished" podID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerID="ba8a4fa6b882d63b72436edee5bea72f1d4cff677932c7d5362524985ccb3d47" exitCode=60 Oct 31 22:00:04.749835 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.749801 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerDied","Data":"ba8a4fa6b882d63b72436edee5bea72f1d4cff677932c7d5362524985ccb3d47"} Oct 31 22:00:04.750047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.750018 4294 scope.go:122] "RemoveContainer" containerID="ba8a4fa6b882d63b72436edee5bea72f1d4cff677932c7d5362524985ccb3d47" Oct 31 22:00:04.750426 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:04.750365 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b_odh-ai-gateway-infra(ad066037-9e1a-48be-9db0-f3a9f8768a6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" Oct 31 22:00:04.752625 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.752597 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerStarted","Data":"529ad599a4c302e593a5807d74375c7472bd19a8cb3cdc4909d17e5b02aefb96"} Oct 31 22:00:04.756400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.756376 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerStarted","Data":"ac2630b2023613e6f6f3a845563c9923d78980844b5c0feca740d79aca7bfcc9"} Oct 31 22:00:04.757873 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.757855 4294 scope.go:122] "RemoveContainer" containerID="06190620932f4d99ba549a3520790ce4f5964d58fcf56bfc46918252a35622b0" Oct 31 22:00:04.758521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.758499 4294 generic.go:356] "Generic (PLEG): container finished" podID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerID="a7475924cf0353cd252a9d167476ff533f798c4c030c225369a88ecc04f5105b" exitCode=60 Oct 31 22:00:04.758625 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.758605 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" event={"ID":"e9b968b4-b2d0-4bc7-a8c1-d58735388295","Type":"ContainerDied","Data":"a7475924cf0353cd252a9d167476ff533f798c4c030c225369a88ecc04f5105b"} Oct 31 22:00:04.758837 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.758776 4294 scope.go:122] "RemoveContainer" containerID="a7475924cf0353cd252a9d167476ff533f798c4c030c225369a88ecc04f5105b" Oct 31 22:00:04.759072 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:04.759050 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr_odh-ai-gateway-infra(e9b968b4-b2d0-4bc7-a8c1-d58735388295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" Oct 31 22:00:04.760620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.760599 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerStarted","Data":"633532969aeeae7382da2b3bf395b2726863596669babcdb85a979f8538ab6dc"} Oct 31 22:00:04.762313 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.762279 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerStarted","Data":"61af786c12e6a24b63bc964877ffa965bea5952bc552aabab167a42dbc8cc7ab"} Oct 31 22:00:04.764018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.763998 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerStarted","Data":"8d4520779f8c1c3ea61e775146838e77d40a540b243beb9f3a980e18211c2714"} Oct 31 22:00:04.765814 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.765791 4294 generic.go:356] "Generic (PLEG): container finished" podID="95452683-4ed9-4672-9026-339d1a0a07e5" containerID="e945d36dfca1e986c98ce484d50d4b095235dd5f186a008904b6c9c00b8f79b0" exitCode=60 Oct 31 22:00:04.765908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.765863 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" event={"ID":"95452683-4ed9-4672-9026-339d1a0a07e5","Type":"ContainerDied","Data":"e945d36dfca1e986c98ce484d50d4b095235dd5f186a008904b6c9c00b8f79b0"} Oct 31 22:00:04.766079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.766063 4294 scope.go:122] "RemoveContainer" containerID="e945d36dfca1e986c98ce484d50d4b095235dd5f186a008904b6c9c00b8f79b0" Oct 31 22:00:04.766287 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:04.766264 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp_odh-ai-gateway-infra(95452683-4ed9-4672-9026-339d1a0a07e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" Oct 31 22:00:04.767989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.767965 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerStarted","Data":"cedbba0ab772cc4ce3bac62a01aa5a628764c8499ce26a2dedbf382449827600"} Oct 31 22:00:04.770048 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.770028 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" event={"ID":"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713","Type":"ContainerStarted","Data":"7bed58d4a3e15904d862627adb8b4e1b3cabcb904fc367c685457bec8453d5ae"} Oct 31 22:00:04.775489 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:04.775467 4294 scope.go:122] "RemoveContainer" containerID="154e84ff215a075d5b8807963edf0168fa5b43d82bd0f56b2018ba4740c5b208" Oct 31 22:00:05.302028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.302001 4294 scope.go:122] "RemoveContainer" containerID="ac71c65aee79a12e83bc3df53645292c23afc79dc312df96a09a2992d3630ccd" Oct 31 22:00:05.789703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.789673 4294 generic.go:356] "Generic (PLEG): container finished" podID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerID="ac2630b2023613e6f6f3a845563c9923d78980844b5c0feca740d79aca7bfcc9" exitCode=60 Oct 31 22:00:05.790162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.789761 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerDied","Data":"ac2630b2023613e6f6f3a845563c9923d78980844b5c0feca740d79aca7bfcc9"} Oct 31 22:00:05.790162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.789847 4294 scope.go:122] "RemoveContainer" containerID="406a7f39ccbef3bd3a82ef99d1eb47cf09791c01fe4abd299bc6b0bcdf26fad0" Oct 31 22:00:05.790162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.790028 4294 scope.go:122] "RemoveContainer" containerID="ac2630b2023613e6f6f3a845563c9923d78980844b5c0feca740d79aca7bfcc9" Oct 31 22:00:05.790423 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.790397 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5_odh-ai-gateway-infra(decb8099-9e1b-4e62-8641-9769c2c9c83d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" Oct 31 22:00:05.791909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.791889 4294 scope.go:122] "RemoveContainer" containerID="a7475924cf0353cd252a9d167476ff533f798c4c030c225369a88ecc04f5105b" Oct 31 22:00:05.792186 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.792169 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr_odh-ai-gateway-infra(e9b968b4-b2d0-4bc7-a8c1-d58735388295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" Oct 31 22:00:05.793490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.793467 4294 generic.go:356] "Generic (PLEG): container finished" podID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerID="633532969aeeae7382da2b3bf395b2726863596669babcdb85a979f8538ab6dc" exitCode=60 Oct 31 22:00:05.793562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.793543 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerDied","Data":"633532969aeeae7382da2b3bf395b2726863596669babcdb85a979f8538ab6dc"} Oct 31 22:00:05.793777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.793760 4294 scope.go:122] "RemoveContainer" containerID="633532969aeeae7382da2b3bf395b2726863596669babcdb85a979f8538ab6dc" Oct 31 22:00:05.794046 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.794020 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j_odh-ai-gateway-infra(86ef7678-8bf6-443c-90e1-8c1af8de24d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" Oct 31 22:00:05.795435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.795414 4294 generic.go:356] "Generic (PLEG): container finished" podID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerID="61af786c12e6a24b63bc964877ffa965bea5952bc552aabab167a42dbc8cc7ab" exitCode=60 Oct 31 22:00:05.795528 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.795485 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerDied","Data":"61af786c12e6a24b63bc964877ffa965bea5952bc552aabab167a42dbc8cc7ab"} Oct 31 22:00:05.795726 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.795712 4294 scope.go:122] "RemoveContainer" containerID="61af786c12e6a24b63bc964877ffa965bea5952bc552aabab167a42dbc8cc7ab" Oct 31 22:00:05.795951 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.795931 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv_odh-ai-gateway-infra(46a37b22-93fb-4781-a126-6a138e8ba5c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" Oct 31 22:00:05.797498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.797475 4294 generic.go:356] "Generic (PLEG): container finished" podID="2237055a-125a-45a4-998f-1ce799325f97" containerID="8d4520779f8c1c3ea61e775146838e77d40a540b243beb9f3a980e18211c2714" exitCode=60 Oct 31 22:00:05.797598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.797540 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerDied","Data":"8d4520779f8c1c3ea61e775146838e77d40a540b243beb9f3a980e18211c2714"} Oct 31 22:00:05.797748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.797735 4294 scope.go:122] "RemoveContainer" containerID="8d4520779f8c1c3ea61e775146838e77d40a540b243beb9f3a980e18211c2714" Oct 31 22:00:05.797993 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.797969 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964_odh-ai-gateway-infra(2237055a-125a-45a4-998f-1ce799325f97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" podUID="2237055a-125a-45a4-998f-1ce799325f97" Oct 31 22:00:05.799591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.799573 4294 scope.go:122] "RemoveContainer" containerID="e945d36dfca1e986c98ce484d50d4b095235dd5f186a008904b6c9c00b8f79b0" Oct 31 22:00:05.799845 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.799818 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp_odh-ai-gateway-infra(95452683-4ed9-4672-9026-339d1a0a07e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" Oct 31 22:00:05.801142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.801120 4294 generic.go:356] "Generic (PLEG): container finished" podID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerID="cedbba0ab772cc4ce3bac62a01aa5a628764c8499ce26a2dedbf382449827600" exitCode=60 Oct 31 22:00:05.801259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.801201 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerDied","Data":"cedbba0ab772cc4ce3bac62a01aa5a628764c8499ce26a2dedbf382449827600"} Oct 31 22:00:05.801699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.801651 4294 scope.go:122] "RemoveContainer" containerID="cedbba0ab772cc4ce3bac62a01aa5a628764c8499ce26a2dedbf382449827600" Oct 31 22:00:05.801938 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.801907 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb_odh-ai-gateway-infra(0115189e-cf8e-4af0-b4d6-da4d82bd9db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" Oct 31 22:00:05.803433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.803413 4294 generic.go:356] "Generic (PLEG): container finished" podID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerID="7bed58d4a3e15904d862627adb8b4e1b3cabcb904fc367c685457bec8453d5ae" exitCode=60 Oct 31 22:00:05.803527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.803492 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" event={"ID":"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713","Type":"ContainerDied","Data":"7bed58d4a3e15904d862627adb8b4e1b3cabcb904fc367c685457bec8453d5ae"} Oct 31 22:00:05.803678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.803664 4294 scope.go:122] "RemoveContainer" containerID="7bed58d4a3e15904d862627adb8b4e1b3cabcb904fc367c685457bec8453d5ae" Oct 31 22:00:05.803903 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.803876 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2_odh-ai-gateway-infra(6e9ff7be-fa18-4a26-83fc-8e6f88ff1713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" Oct 31 22:00:05.805407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.805386 4294 scope.go:122] "RemoveContainer" containerID="d72f659f125bcea9a363dc623881bcf7c45f0ae25116407f36f805d79393d79e" Oct 31 22:00:05.805607 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.805582 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h_odh-ai-gateway-infra(ec3be5fe-e715-4121-bd7b-e76b9fb6edbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" Oct 31 22:00:05.805999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.805983 4294 scope.go:122] "RemoveContainer" containerID="9a99d72ce517d991bb12d1f29d3e82a874a5b4adf70bb3b1145a30ed4917456f" Oct 31 22:00:05.806986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.806966 4294 generic.go:356] "Generic (PLEG): container finished" podID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerID="26484b008efef826e1f9f89c1e0c101f9c86107824e8cbf99f31f389e17e4b01" exitCode=60 Oct 31 22:00:05.807068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.807054 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerDied","Data":"26484b008efef826e1f9f89c1e0c101f9c86107824e8cbf99f31f389e17e4b01"} Oct 31 22:00:05.807464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.807440 4294 scope.go:122] "RemoveContainer" containerID="26484b008efef826e1f9f89c1e0c101f9c86107824e8cbf99f31f389e17e4b01" Oct 31 22:00:05.807701 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.807676 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp_odh-ai-gateway-infra(8994495f-2a50-4a8b-af81-e5c14c57c4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" Oct 31 22:00:05.808886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.808867 4294 generic.go:356] "Generic (PLEG): container finished" podID="0b742073-6765-4c13-931d-245b538180f6" containerID="5109bb6c8e15394b01d8d3c81d51390758f60166b611881c5927e1feb2e41486" exitCode=60 Oct 31 22:00:05.808980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.808945 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerDied","Data":"5109bb6c8e15394b01d8d3c81d51390758f60166b611881c5927e1feb2e41486"} Oct 31 22:00:05.809125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.809108 4294 scope.go:122] "RemoveContainer" containerID="5109bb6c8e15394b01d8d3c81d51390758f60166b611881c5927e1feb2e41486" Oct 31 22:00:05.809385 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.809367 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w_odh-ai-gateway-infra(0b742073-6765-4c13-931d-245b538180f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" podUID="0b742073-6765-4c13-931d-245b538180f6" Oct 31 22:00:05.810703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.810684 4294 scope.go:122] "RemoveContainer" containerID="ba8a4fa6b882d63b72436edee5bea72f1d4cff677932c7d5362524985ccb3d47" Oct 31 22:00:05.810923 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.810899 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b_odh-ai-gateway-infra(ad066037-9e1a-48be-9db0-f3a9f8768a6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" Oct 31 22:00:05.812473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.812450 4294 generic.go:356] "Generic (PLEG): container finished" podID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerID="529ad599a4c302e593a5807d74375c7472bd19a8cb3cdc4909d17e5b02aefb96" exitCode=60 Oct 31 22:00:05.812575 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.812480 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerDied","Data":"529ad599a4c302e593a5807d74375c7472bd19a8cb3cdc4909d17e5b02aefb96"} Oct 31 22:00:05.812734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.812718 4294 scope.go:122] "RemoveContainer" containerID="529ad599a4c302e593a5807d74375c7472bd19a8cb3cdc4909d17e5b02aefb96" Oct 31 22:00:05.812944 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:05.812925 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4_odh-ai-gateway-infra(6f793472-466a-4b0e-a83f-f21aa9d01c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" Oct 31 22:00:05.820067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.820048 4294 scope.go:122] "RemoveContainer" containerID="c38e581f0906df474f814eb515386044100f4a7299b13fd947f49636e17999b2" Oct 31 22:00:05.834444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.834422 4294 scope.go:122] "RemoveContainer" containerID="bec3c6ffa5765bc8c1102ecb2b285000780f9b7a65fca1829959ab80bda8a55a" Oct 31 22:00:05.853263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.853241 4294 scope.go:122] "RemoveContainer" containerID="509ec8b4c31b35fc7cda1ff6dfc8714d03aa67e7da2bbfcd13bcc82aca4cc5ac" Oct 31 22:00:05.866974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.866955 4294 scope.go:122] "RemoveContainer" containerID="7d549a95cf746c4e03851d6a163e87a81aa9ac1278e861fd761ccf4c6a294f20" Oct 31 22:00:05.881572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.881545 4294 scope.go:122] "RemoveContainer" containerID="8900cc7705aa47e712412d57bdbebc944b5b31c045c8839fc10957abcfcc08ec" Oct 31 22:00:05.911400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.911373 4294 scope.go:122] "RemoveContainer" containerID="f89c601f883c0177072e76ea28ed55df0842281e81f69858b4be4b41b51d8192" Oct 31 22:00:05.928944 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:05.928919 4294 scope.go:122] "RemoveContainer" containerID="712d50fe9ce69dc3c221cb869f8cf4bc1057f00e0800b109da5fcca493532ef6" Oct 31 22:00:06.821613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.821581 4294 scope.go:122] "RemoveContainer" containerID="7bed58d4a3e15904d862627adb8b4e1b3cabcb904fc367c685457bec8453d5ae" Oct 31 22:00:06.822081 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.821972 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2_odh-ai-gateway-infra(6e9ff7be-fa18-4a26-83fc-8e6f88ff1713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" Oct 31 22:00:06.823391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.823364 4294 scope.go:122] "RemoveContainer" containerID="26484b008efef826e1f9f89c1e0c101f9c86107824e8cbf99f31f389e17e4b01" Oct 31 22:00:06.823685 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.823661 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp_odh-ai-gateway-infra(8994495f-2a50-4a8b-af81-e5c14c57c4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" Oct 31 22:00:06.825107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.825089 4294 scope.go:122] "RemoveContainer" containerID="5109bb6c8e15394b01d8d3c81d51390758f60166b611881c5927e1feb2e41486" Oct 31 22:00:06.825406 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.825383 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w_odh-ai-gateway-infra(0b742073-6765-4c13-931d-245b538180f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" podUID="0b742073-6765-4c13-931d-245b538180f6" Oct 31 22:00:06.826699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.826684 4294 scope.go:122] "RemoveContainer" containerID="529ad599a4c302e593a5807d74375c7472bd19a8cb3cdc4909d17e5b02aefb96" Oct 31 22:00:06.826923 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.826901 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4_odh-ai-gateway-infra(6f793472-466a-4b0e-a83f-f21aa9d01c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" Oct 31 22:00:06.828364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.828343 4294 scope.go:122] "RemoveContainer" containerID="ac2630b2023613e6f6f3a845563c9923d78980844b5c0feca740d79aca7bfcc9" Oct 31 22:00:06.828596 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.828571 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5_odh-ai-gateway-infra(decb8099-9e1b-4e62-8641-9769c2c9c83d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" Oct 31 22:00:06.829996 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.829978 4294 scope.go:122] "RemoveContainer" containerID="633532969aeeae7382da2b3bf395b2726863596669babcdb85a979f8538ab6dc" Oct 31 22:00:06.830198 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.830178 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j_odh-ai-gateway-infra(86ef7678-8bf6-443c-90e1-8c1af8de24d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" Oct 31 22:00:06.831599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.831583 4294 scope.go:122] "RemoveContainer" containerID="61af786c12e6a24b63bc964877ffa965bea5952bc552aabab167a42dbc8cc7ab" Oct 31 22:00:06.831807 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.831785 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv_odh-ai-gateway-infra(46a37b22-93fb-4781-a126-6a138e8ba5c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" Oct 31 22:00:06.833088 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.833070 4294 scope.go:122] "RemoveContainer" containerID="8d4520779f8c1c3ea61e775146838e77d40a540b243beb9f3a980e18211c2714" Oct 31 22:00:06.833330 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.833286 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964_odh-ai-gateway-infra(2237055a-125a-45a4-998f-1ce799325f97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" podUID="2237055a-125a-45a4-998f-1ce799325f97" Oct 31 22:00:06.834624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:06.834607 4294 scope.go:122] "RemoveContainer" containerID="cedbba0ab772cc4ce3bac62a01aa5a628764c8499ce26a2dedbf382449827600" Oct 31 22:00:06.834801 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:06.834785 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb_odh-ai-gateway-infra(0115189e-cf8e-4af0-b4d6-da4d82bd9db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" Oct 31 22:00:10.952750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:10.952710 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq"] Oct 31 22:00:10.954490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:10.954460 4294 cpu_manager.go:414] "containerMap: removing container" podUID="609de17a-3818-49a4-a0c6-3edc99e4337d" containerName="revoke-keys" Oct 31 22:00:10.954704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:10.954671 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="609de17a-3818-49a4-a0c6-3edc99e4337d" containerName="revoke-keys" Oct 31 22:00:10.955014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:10.954968 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="609de17a-3818-49a4-a0c6-3edc99e4337d" containerName="revoke-keys" Oct 31 22:00:11.032751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.032717 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq"] Oct 31 22:00:11.043657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.034117 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.120711 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.120672 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.125009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.124005 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/0a44c138-1a2c-4bb9-b57c-32155db4050e-kube-api-access-qct68\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.125009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.124137 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.125009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.121398 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:00:11.225843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.225757 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.225843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.225814 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/0a44c138-1a2c-4bb9-b57c-32155db4050e-kube-api-access-qct68\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.226050 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.225877 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.226571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.226549 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.228145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.228121 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.249279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.249246 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/0a44c138-1a2c-4bb9-b57c-32155db4050e-kube-api-access-qct68\") pod \"maas-api-revoke-sub-f715b9dd-bgljq\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.436966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.436936 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:11.738914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.738879 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq"] Oct 31 22:00:11.739035 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:11.738955 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a44c138_1a2c_4bb9_b57c_32155db4050e.slice/crio-af0059ea0a6ea7d33223335887056721b9818fd440f6573114e9d65addd3ba87 WatchSource:0}: Error finding container af0059ea0a6ea7d33223335887056721b9818fd440f6573114e9d65addd3ba87: Status 404 returned error can't find the container with id af0059ea0a6ea7d33223335887056721b9818fd440f6573114e9d65addd3ba87 Oct 31 22:00:11.875685 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:11.875649 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" event={"ID":"0a44c138-1a2c-4bb9-b57c-32155db4050e","Type":"ContainerStarted","Data":"af0059ea0a6ea7d33223335887056721b9818fd440f6573114e9d65addd3ba87"} Oct 31 22:00:12.884894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:12.884862 4294 generic.go:356] "Generic (PLEG): container finished" podID="0a44c138-1a2c-4bb9-b57c-32155db4050e" containerID="6ba6aac779557d7250ee2e47d41262643aa872c09de8ea86fca5f51a7335edae" exitCode=0 Oct 31 22:00:12.885284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:12.884940 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" event={"ID":"0a44c138-1a2c-4bb9-b57c-32155db4050e","Type":"ContainerDied","Data":"6ba6aac779557d7250ee2e47d41262643aa872c09de8ea86fca5f51a7335edae"} Oct 31 22:00:14.049729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.049702 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:14.067692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.067663 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-xks-ca\") pod \"0a44c138-1a2c-4bb9-b57c-32155db4050e\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " Oct 31 22:00:14.067818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.067808 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a44c138-1a2c-4bb9-b57c-32155db4050e-kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/0a44c138-1a2c-4bb9-b57c-32155db4050e-kube-api-access-qct68\") pod \"0a44c138-1a2c-4bb9-b57c-32155db4050e\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " Oct 31 22:00:14.067963 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.067949 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-openshift-ca\") pod \"0a44c138-1a2c-4bb9-b57c-32155db4050e\" (UID: \"0a44c138-1a2c-4bb9-b57c-32155db4050e\") " Oct 31 22:00:14.068323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.068279 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-openshift-ca" pod "0a44c138-1a2c-4bb9-b57c-32155db4050e" (UID: "0a44c138-1a2c-4bb9-b57c-32155db4050e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:14.068481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.068463 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:14.069854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.069835 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-xks-ca" pod "0a44c138-1a2c-4bb9-b57c-32155db4050e" (UID: "0a44c138-1a2c-4bb9-b57c-32155db4050e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:00:14.069947 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.069864 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a44c138-1a2c-4bb9-b57c-32155db4050e-kube-api-access-qct68" pod "0a44c138-1a2c-4bb9-b57c-32155db4050e" (UID: "0a44c138-1a2c-4bb9-b57c-32155db4050e"). InnerVolumeSpecName "kube-api-access-qct68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:14.169596 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.169519 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qct68\" (UniqueName: \"kubernetes.io/projected/0a44c138-1a2c-4bb9-b57c-32155db4050e-kube-api-access-qct68\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:14.169596 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.169544 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a44c138-1a2c-4bb9-b57c-32155db4050e-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:14.902543 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.902512 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" Oct 31 22:00:14.902543 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.902522 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq" event={"ID":"0a44c138-1a2c-4bb9-b57c-32155db4050e","Type":"ContainerDied","Data":"af0059ea0a6ea7d33223335887056721b9818fd440f6573114e9d65addd3ba87"} Oct 31 22:00:14.902739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.902598 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af0059ea0a6ea7d33223335887056721b9818fd440f6573114e9d65addd3ba87" Oct 31 22:00:14.978455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.978431 4294 scope.go:122] "RemoveContainer" containerID="a7475924cf0353cd252a9d167476ff533f798c4c030c225369a88ecc04f5105b" Oct 31 22:00:14.978598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:14.978578 4294 scope.go:122] "RemoveContainer" containerID="e945d36dfca1e986c98ce484d50d4b095235dd5f186a008904b6c9c00b8f79b0" Oct 31 22:00:15.911066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.911039 4294 generic.go:356] "Generic (PLEG): container finished" podID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerID="1495b7831a5992ca2262f2c93117af47e396cdcad9331ccef0817596d4fcf0ee" exitCode=60 Oct 31 22:00:15.911501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.911109 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" event={"ID":"e9b968b4-b2d0-4bc7-a8c1-d58735388295","Type":"ContainerDied","Data":"1495b7831a5992ca2262f2c93117af47e396cdcad9331ccef0817596d4fcf0ee"} Oct 31 22:00:15.911501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.911169 4294 scope.go:122] "RemoveContainer" containerID="a7475924cf0353cd252a9d167476ff533f798c4c030c225369a88ecc04f5105b" Oct 31 22:00:15.911657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.911640 4294 scope.go:122] "RemoveContainer" containerID="1495b7831a5992ca2262f2c93117af47e396cdcad9331ccef0817596d4fcf0ee" Oct 31 22:00:15.911985 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:15.911964 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr_odh-ai-gateway-infra(e9b968b4-b2d0-4bc7-a8c1-d58735388295)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" Oct 31 22:00:15.913228 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.913209 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" event={"ID":"95452683-4ed9-4672-9026-339d1a0a07e5","Type":"ContainerStarted","Data":"d40971c48efbaec35580daa4159e9663f9e38c9a88d66a1a0a57db76f98fab06"} Oct 31 22:00:15.978446 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.978410 4294 scope.go:122] "RemoveContainer" containerID="529ad599a4c302e593a5807d74375c7472bd19a8cb3cdc4909d17e5b02aefb96" Oct 31 22:00:15.978680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.978655 4294 scope.go:122] "RemoveContainer" containerID="633532969aeeae7382da2b3bf395b2726863596669babcdb85a979f8538ab6dc" Oct 31 22:00:15.978799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.978723 4294 scope.go:122] "RemoveContainer" containerID="ba8a4fa6b882d63b72436edee5bea72f1d4cff677932c7d5362524985ccb3d47" Oct 31 22:00:15.978880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:15.978832 4294 scope.go:122] "RemoveContainer" containerID="ac2630b2023613e6f6f3a845563c9923d78980844b5c0feca740d79aca7bfcc9" Oct 31 22:00:16.924043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.924003 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerStarted","Data":"1e5fb394f0c4d2aebbcb0b77d4663e6d59a87f148c8de8b704e7d5c58126c86a"} Oct 31 22:00:16.925895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.925870 4294 generic.go:356] "Generic (PLEG): container finished" podID="95452683-4ed9-4672-9026-339d1a0a07e5" containerID="d40971c48efbaec35580daa4159e9663f9e38c9a88d66a1a0a57db76f98fab06" exitCode=60 Oct 31 22:00:16.926014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.925947 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" event={"ID":"95452683-4ed9-4672-9026-339d1a0a07e5","Type":"ContainerDied","Data":"d40971c48efbaec35580daa4159e9663f9e38c9a88d66a1a0a57db76f98fab06"} Oct 31 22:00:16.926014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.926008 4294 scope.go:122] "RemoveContainer" containerID="e945d36dfca1e986c98ce484d50d4b095235dd5f186a008904b6c9c00b8f79b0" Oct 31 22:00:16.926362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.926347 4294 scope.go:122] "RemoveContainer" containerID="d40971c48efbaec35580daa4159e9663f9e38c9a88d66a1a0a57db76f98fab06" Oct 31 22:00:16.926665 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:16.926632 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp_odh-ai-gateway-infra(95452683-4ed9-4672-9026-339d1a0a07e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" Oct 31 22:00:16.927908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.927881 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerStarted","Data":"3610120a95a11749cc29019a3fcbab19a22a62779081ae7a50640b0b53efb8a6"} Oct 31 22:00:16.929611 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.929591 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerStarted","Data":"bdf10a3fca70b2c7371b67e61f1ad81936d86514496642b4ab0d91ffc8b50e91"} Oct 31 22:00:16.931431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.931410 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerStarted","Data":"afb0f6c1a1617a4bfaf977b49e4b0078aa905cc4849b57af55830d3aff5a7258"} Oct 31 22:00:16.963845 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.963810 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr"] Oct 31 22:00:16.977797 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.977756 4294 scope.go:122] "RemoveContainer" containerID="d72f659f125bcea9a363dc623881bcf7c45f0ae25116407f36f805d79393d79e" Oct 31 22:00:16.977946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:16.977919 4294 scope.go:122] "RemoveContainer" containerID="7bed58d4a3e15904d862627adb8b4e1b3cabcb904fc367c685457bec8453d5ae" Oct 31 22:00:17.011470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.011442 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp"] Oct 31 22:00:17.318244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.318219 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:17.426359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.426256 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9b968b4-b2d0-4bc7-a8c1-d58735388295-kube-api-access-xvfzn\" (UniqueName: \"kubernetes.io/projected/e9b968b4-b2d0-4bc7-a8c1-d58735388295-kube-api-access-xvfzn\") pod \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " Oct 31 22:00:17.426524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.426441 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9b968b4-b2d0-4bc7-a8c1-d58735388295-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b968b4-b2d0-4bc7-a8c1-d58735388295-maas-api-service-ca\") pod \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\" (UID: \"e9b968b4-b2d0-4bc7-a8c1-d58735388295\") " Oct 31 22:00:17.426798 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.426770 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9b968b4-b2d0-4bc7-a8c1-d58735388295-maas-api-service-ca" pod "e9b968b4-b2d0-4bc7-a8c1-d58735388295" (UID: "e9b968b4-b2d0-4bc7-a8c1-d58735388295"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:17.426914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.426888 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9b968b4-b2d0-4bc7-a8c1-d58735388295-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:17.428610 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.428582 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9b968b4-b2d0-4bc7-a8c1-d58735388295-kube-api-access-xvfzn" pod "e9b968b4-b2d0-4bc7-a8c1-d58735388295" (UID: "e9b968b4-b2d0-4bc7-a8c1-d58735388295"). InnerVolumeSpecName "kube-api-access-xvfzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:17.527999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.527965 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvfzn\" (UniqueName: \"kubernetes.io/projected/e9b968b4-b2d0-4bc7-a8c1-d58735388295-kube-api-access-xvfzn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:17.950184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.950152 4294 generic.go:356] "Generic (PLEG): container finished" podID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerID="bdf10a3fca70b2c7371b67e61f1ad81936d86514496642b4ab0d91ffc8b50e91" exitCode=60 Oct 31 22:00:17.950700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.950235 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerDied","Data":"bdf10a3fca70b2c7371b67e61f1ad81936d86514496642b4ab0d91ffc8b50e91"} Oct 31 22:00:17.950700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.950359 4294 scope.go:122] "RemoveContainer" containerID="529ad599a4c302e593a5807d74375c7472bd19a8cb3cdc4909d17e5b02aefb96" Oct 31 22:00:17.950818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.950711 4294 scope.go:122] "RemoveContainer" containerID="bdf10a3fca70b2c7371b67e61f1ad81936d86514496642b4ab0d91ffc8b50e91" Oct 31 22:00:17.951063 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:17.951038 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4_odh-ai-gateway-infra(6f793472-466a-4b0e-a83f-f21aa9d01c92)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" Oct 31 22:00:17.952621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.952602 4294 generic.go:356] "Generic (PLEG): container finished" podID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerID="afb0f6c1a1617a4bfaf977b49e4b0078aa905cc4849b57af55830d3aff5a7258" exitCode=60 Oct 31 22:00:17.952737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.952653 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerDied","Data":"afb0f6c1a1617a4bfaf977b49e4b0078aa905cc4849b57af55830d3aff5a7258"} Oct 31 22:00:17.952983 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.952969 4294 scope.go:122] "RemoveContainer" containerID="afb0f6c1a1617a4bfaf977b49e4b0078aa905cc4849b57af55830d3aff5a7258" Oct 31 22:00:17.953256 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:17.953237 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5_odh-ai-gateway-infra(decb8099-9e1b-4e62-8641-9769c2c9c83d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" Oct 31 22:00:17.954066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.954048 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" Oct 31 22:00:17.954154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.954052 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr" event={"ID":"e9b968b4-b2d0-4bc7-a8c1-d58735388295","Type":"ContainerDied","Data":"b20b25ce6d7431c48914d825ca240313c3b89176be73178dcc52b11d00425c4e"} Oct 31 22:00:17.955859 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.955841 4294 generic.go:356] "Generic (PLEG): container finished" podID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerID="1e5fb394f0c4d2aebbcb0b77d4663e6d59a87f148c8de8b704e7d5c58126c86a" exitCode=60 Oct 31 22:00:17.955963 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.955914 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerDied","Data":"1e5fb394f0c4d2aebbcb0b77d4663e6d59a87f148c8de8b704e7d5c58126c86a"} Oct 31 22:00:17.956283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.956266 4294 scope.go:122] "RemoveContainer" containerID="1e5fb394f0c4d2aebbcb0b77d4663e6d59a87f148c8de8b704e7d5c58126c86a" Oct 31 22:00:17.956628 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:17.956604 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j_odh-ai-gateway-infra(86ef7678-8bf6-443c-90e1-8c1af8de24d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" Oct 31 22:00:17.959769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.959749 4294 generic.go:356] "Generic (PLEG): container finished" podID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerID="de478d8a1722ba6fd1888837b88bbb824684b0e48851b0b19f9aace577e68a15" exitCode=60 Oct 31 22:00:17.959855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.959842 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" event={"ID":"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713","Type":"ContainerDied","Data":"de478d8a1722ba6fd1888837b88bbb824684b0e48851b0b19f9aace577e68a15"} Oct 31 22:00:17.960138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.960120 4294 scope.go:122] "RemoveContainer" containerID="de478d8a1722ba6fd1888837b88bbb824684b0e48851b0b19f9aace577e68a15" Oct 31 22:00:17.960370 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:17.960350 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2_odh-ai-gateway-infra(6e9ff7be-fa18-4a26-83fc-8e6f88ff1713)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" Oct 31 22:00:17.962185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.962163 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" event={"ID":"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd","Type":"ContainerStarted","Data":"784112f6c0e5415923c808622a6ee967b2e09f32020063d85cb63b4d141ec93b"} Oct 31 22:00:17.965099 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.965080 4294 generic.go:356] "Generic (PLEG): container finished" podID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerID="3610120a95a11749cc29019a3fcbab19a22a62779081ae7a50640b0b53efb8a6" exitCode=60 Oct 31 22:00:17.965190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.965106 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerDied","Data":"3610120a95a11749cc29019a3fcbab19a22a62779081ae7a50640b0b53efb8a6"} Oct 31 22:00:17.965461 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.965442 4294 scope.go:122] "RemoveContainer" containerID="3610120a95a11749cc29019a3fcbab19a22a62779081ae7a50640b0b53efb8a6" Oct 31 22:00:17.965761 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:17.965740 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b_odh-ai-gateway-infra(ad066037-9e1a-48be-9db0-f3a9f8768a6b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" Oct 31 22:00:17.966534 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.966517 4294 scope.go:122] "RemoveContainer" containerID="ac2630b2023613e6f6f3a845563c9923d78980844b5c0feca740d79aca7bfcc9" Oct 31 22:00:17.977858 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.977838 4294 scope.go:122] "RemoveContainer" containerID="26484b008efef826e1f9f89c1e0c101f9c86107824e8cbf99f31f389e17e4b01" Oct 31 22:00:17.979673 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.979624 4294 scope.go:122] "RemoveContainer" containerID="61af786c12e6a24b63bc964877ffa965bea5952bc552aabab167a42dbc8cc7ab" Oct 31 22:00:17.980498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.980416 4294 scope.go:122] "RemoveContainer" containerID="5109bb6c8e15394b01d8d3c81d51390758f60166b611881c5927e1feb2e41486" Oct 31 22:00:17.980591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:17.980555 4294 scope.go:122] "RemoveContainer" containerID="8d4520779f8c1c3ea61e775146838e77d40a540b243beb9f3a980e18211c2714" Oct 31 22:00:18.000680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.000649 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j"] Oct 31 22:00:18.022906 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.022879 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b"] Oct 31 22:00:18.043339 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:18.041493 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-192c5e9d9ff4: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:18.043339 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:18.041524 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:18.043339 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:18.041565 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials podName:c977fc93-3845-44c2-a8b8-3bdafbcd2656 nodeName:}" failed. No retries permitted until 2026-10-31 22:00:18.541551565 +0000 UTC m=+2658.666658774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656") : secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:18.094937 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.094905 4294 scope.go:122] "RemoveContainer" containerID="1495b7831a5992ca2262f2c93117af47e396cdcad9331ccef0817596d4fcf0ee" Oct 31 22:00:18.110871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.110839 4294 scope.go:122] "RemoveContainer" containerID="633532969aeeae7382da2b3bf395b2726863596669babcdb85a979f8538ab6dc" Oct 31 22:00:18.112467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.112445 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5"] Oct 31 22:00:18.130854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.130833 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:18.134408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.134389 4294 scope.go:122] "RemoveContainer" containerID="7bed58d4a3e15904d862627adb8b4e1b3cabcb904fc367c685457bec8453d5ae" Oct 31 22:00:18.156111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.156086 4294 scope.go:122] "RemoveContainer" containerID="ba8a4fa6b882d63b72436edee5bea72f1d4cff677932c7d5362524985ccb3d47" Oct 31 22:00:18.156888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.156861 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4"] Oct 31 22:00:18.211396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.211366 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr"] Oct 31 22:00:18.223591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.223560 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891400-lzvkr"] Oct 31 22:00:18.242942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.242916 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95452683-4ed9-4672-9026-339d1a0a07e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95452683-4ed9-4672-9026-339d1a0a07e5-maas-api-service-ca\") pod \"95452683-4ed9-4672-9026-339d1a0a07e5\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " Oct 31 22:00:18.243170 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.243129 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95452683-4ed9-4672-9026-339d1a0a07e5-kube-api-access-blgln\" (UniqueName: \"kubernetes.io/projected/95452683-4ed9-4672-9026-339d1a0a07e5-kube-api-access-blgln\") pod \"95452683-4ed9-4672-9026-339d1a0a07e5\" (UID: \"95452683-4ed9-4672-9026-339d1a0a07e5\") " Oct 31 22:00:18.243369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.243346 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95452683-4ed9-4672-9026-339d1a0a07e5-maas-api-service-ca" pod "95452683-4ed9-4672-9026-339d1a0a07e5" (UID: "95452683-4ed9-4672-9026-339d1a0a07e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:18.243711 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.243693 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/95452683-4ed9-4672-9026-339d1a0a07e5-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:18.273464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.273428 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95452683-4ed9-4672-9026-339d1a0a07e5-kube-api-access-blgln" pod "95452683-4ed9-4672-9026-339d1a0a07e5" (UID: "95452683-4ed9-4672-9026-339d1a0a07e5"). InnerVolumeSpecName "kube-api-access-blgln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:18.345260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.345229 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blgln\" (UniqueName: \"kubernetes.io/projected/95452683-4ed9-4672-9026-339d1a0a07e5-kube-api-access-blgln\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:18.548141 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:18.548115 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-192c5e9d9ff4: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:18.548141 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:18.548144 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:18.548381 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:18.548193 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials podName:c977fc93-3845-44c2-a8b8-3bdafbcd2656 nodeName:}" failed. No retries permitted until 2026-10-31 22:00:19.548178878 +0000 UTC m=+2659.673286087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656") : secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:18.978617 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.978588 4294 scope.go:122] "RemoveContainer" containerID="cedbba0ab772cc4ce3bac62a01aa5a628764c8499ce26a2dedbf382449827600" Oct 31 22:00:18.988403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.987671 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerStarted","Data":"5fc17ba051cb96a865d915e82025b15766d7b304ad347ec687846d8253dda139"} Oct 31 22:00:18.990168 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.990141 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerStarted","Data":"486c940b5e9420c10251d12d7f73cb8f68be466ace9e5a9f7021013fd2466128"} Oct 31 22:00:18.991834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.991798 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" event={"ID":"95452683-4ed9-4672-9026-339d1a0a07e5","Type":"ContainerDied","Data":"11e0bbc973ca075da7543a156802e3165efe9ea3bd964243ad00005dff77643b"} Oct 31 22:00:18.991930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.991854 4294 scope.go:122] "RemoveContainer" containerID="d40971c48efbaec35580daa4159e9663f9e38c9a88d66a1a0a57db76f98fab06" Oct 31 22:00:18.991981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:18.991930 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp" Oct 31 22:00:19.007796 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.007768 4294 generic.go:356] "Generic (PLEG): container finished" podID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerID="784112f6c0e5415923c808622a6ee967b2e09f32020063d85cb63b4d141ec93b" exitCode=60 Oct 31 22:00:19.007987 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.007860 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" event={"ID":"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd","Type":"ContainerDied","Data":"784112f6c0e5415923c808622a6ee967b2e09f32020063d85cb63b4d141ec93b"} Oct 31 22:00:19.008336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.008287 4294 scope.go:122] "RemoveContainer" containerID="784112f6c0e5415923c808622a6ee967b2e09f32020063d85cb63b4d141ec93b" Oct 31 22:00:19.008699 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:19.008675 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h_odh-ai-gateway-infra(ec3be5fe-e715-4121-bd7b-e76b9fb6edbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" Oct 31 22:00:19.011786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.011763 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerStarted","Data":"50a256976b9e3830ac65ca093622005558b506794f91b77ae1532cc4bfeea15d"} Oct 31 22:00:19.014226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.014203 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerStarted","Data":"0dbf317db7c05eb004fe55b1bfb9c72372ca5fcba502a094241c64b286e9d16b"} Oct 31 22:00:19.031189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.031161 4294 scope.go:122] "RemoveContainer" containerID="d72f659f125bcea9a363dc623881bcf7c45f0ae25116407f36f805d79393d79e" Oct 31 22:00:19.050363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.046884 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2"] Oct 31 22:00:19.111789 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.110944 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h"] Oct 31 22:00:19.476841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.476800 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp"] Oct 31 22:00:19.480904 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.480872 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4a73b1-29891400-dfhnp"] Oct 31 22:00:19.513552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.513520 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:19.517747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.517725 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:19.521286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.521265 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:19.525530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.525512 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:19.529249 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.529233 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:19.567206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.567177 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ef7678-8bf6-443c-90e1-8c1af8de24d7-kube-api-access-rzcsq\" (UniqueName: \"kubernetes.io/projected/86ef7678-8bf6-443c-90e1-8c1af8de24d7-kube-api-access-rzcsq\") pod \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " Oct 31 22:00:19.567406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.567231 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ef7678-8bf6-443c-90e1-8c1af8de24d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ef7678-8bf6-443c-90e1-8c1af8de24d7-maas-api-service-ca\") pod \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\" (UID: \"86ef7678-8bf6-443c-90e1-8c1af8de24d7\") " Oct 31 22:00:19.567663 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.567639 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ef7678-8bf6-443c-90e1-8c1af8de24d7-maas-api-service-ca" pod "86ef7678-8bf6-443c-90e1-8c1af8de24d7" (UID: "86ef7678-8bf6-443c-90e1-8c1af8de24d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:19.568021 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:19.567983 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-192c5e9d9ff4: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:19.568021 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:19.568017 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:19.568163 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:19.568064 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials podName:c977fc93-3845-44c2-a8b8-3bdafbcd2656 nodeName:}" failed. No retries permitted until 2026-10-31 22:00:21.568048735 +0000 UTC m=+2661.693155944 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656") : secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:19.568163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.567986 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86ef7678-8bf6-443c-90e1-8c1af8de24d7-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.569716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.569694 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ef7678-8bf6-443c-90e1-8c1af8de24d7-kube-api-access-rzcsq" pod "86ef7678-8bf6-443c-90e1-8c1af8de24d7" (UID: "86ef7678-8bf6-443c-90e1-8c1af8de24d7"). InnerVolumeSpecName "kube-api-access-rzcsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:19.669424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669387 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f793472-466a-4b0e-a83f-f21aa9d01c92-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f793472-466a-4b0e-a83f-f21aa9d01c92-maas-api-service-ca\") pod \"6f793472-466a-4b0e-a83f-f21aa9d01c92\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " Oct 31 22:00:19.669605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669531 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-kube-api-access-g2f6q\" (UniqueName: \"kubernetes.io/projected/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-kube-api-access-g2f6q\") pod \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " Oct 31 22:00:19.669605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669576 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f793472-466a-4b0e-a83f-f21aa9d01c92-kube-api-access-sh6lv\" (UniqueName: \"kubernetes.io/projected/6f793472-466a-4b0e-a83f-f21aa9d01c92-kube-api-access-sh6lv\") pod \"6f793472-466a-4b0e-a83f-f21aa9d01c92\" (UID: \"6f793472-466a-4b0e-a83f-f21aa9d01c92\") " Oct 31 22:00:19.669724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669651 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-maas-api-service-ca\") pod \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\" (UID: \"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713\") " Oct 31 22:00:19.669803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669755 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decb8099-9e1b-4e62-8641-9769c2c9c83d-kube-api-access-hmqk2\" (UniqueName: \"kubernetes.io/projected/decb8099-9e1b-4e62-8641-9769c2c9c83d-kube-api-access-hmqk2\") pod \"decb8099-9e1b-4e62-8641-9769c2c9c83d\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " Oct 31 22:00:19.669803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669792 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad066037-9e1a-48be-9db0-f3a9f8768a6b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad066037-9e1a-48be-9db0-f3a9f8768a6b-maas-api-service-ca\") pod \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " Oct 31 22:00:19.669909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669793 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f793472-466a-4b0e-a83f-f21aa9d01c92-maas-api-service-ca" pod "6f793472-466a-4b0e-a83f-f21aa9d01c92" (UID: "6f793472-466a-4b0e-a83f-f21aa9d01c92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:19.669909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669892 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad066037-9e1a-48be-9db0-f3a9f8768a6b-kube-api-access-rr4qv\" (UniqueName: \"kubernetes.io/projected/ad066037-9e1a-48be-9db0-f3a9f8768a6b-kube-api-access-rr4qv\") pod \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\" (UID: \"ad066037-9e1a-48be-9db0-f3a9f8768a6b\") " Oct 31 22:00:19.670009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.669964 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decb8099-9e1b-4e62-8641-9769c2c9c83d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decb8099-9e1b-4e62-8641-9769c2c9c83d-maas-api-service-ca\") pod \"decb8099-9e1b-4e62-8641-9769c2c9c83d\" (UID: \"decb8099-9e1b-4e62-8641-9769c2c9c83d\") " Oct 31 22:00:19.670675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.670647 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzcsq\" (UniqueName: \"kubernetes.io/projected/86ef7678-8bf6-443c-90e1-8c1af8de24d7-kube-api-access-rzcsq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.670787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.670680 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f793472-466a-4b0e-a83f-f21aa9d01c92-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.671043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.671017 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decb8099-9e1b-4e62-8641-9769c2c9c83d-maas-api-service-ca" pod "decb8099-9e1b-4e62-8641-9769c2c9c83d" (UID: "decb8099-9e1b-4e62-8641-9769c2c9c83d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:19.671451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.671426 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad066037-9e1a-48be-9db0-f3a9f8768a6b-maas-api-service-ca" pod "ad066037-9e1a-48be-9db0-f3a9f8768a6b" (UID: "ad066037-9e1a-48be-9db0-f3a9f8768a6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:19.672717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.672689 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-kube-api-access-g2f6q" pod "6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" (UID: "6e9ff7be-fa18-4a26-83fc-8e6f88ff1713"). InnerVolumeSpecName "kube-api-access-g2f6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:19.672830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.672722 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f793472-466a-4b0e-a83f-f21aa9d01c92-kube-api-access-sh6lv" pod "6f793472-466a-4b0e-a83f-f21aa9d01c92" (UID: "6f793472-466a-4b0e-a83f-f21aa9d01c92"). InnerVolumeSpecName "kube-api-access-sh6lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:19.672892 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.672836 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-maas-api-service-ca" pod "6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" (UID: "6e9ff7be-fa18-4a26-83fc-8e6f88ff1713"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:19.673751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.673721 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decb8099-9e1b-4e62-8641-9769c2c9c83d-kube-api-access-hmqk2" pod "decb8099-9e1b-4e62-8641-9769c2c9c83d" (UID: "decb8099-9e1b-4e62-8641-9769c2c9c83d"). InnerVolumeSpecName "kube-api-access-hmqk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:19.674609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.674587 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad066037-9e1a-48be-9db0-f3a9f8768a6b-kube-api-access-rr4qv" pod "ad066037-9e1a-48be-9db0-f3a9f8768a6b" (UID: "ad066037-9e1a-48be-9db0-f3a9f8768a6b"). InnerVolumeSpecName "kube-api-access-rr4qv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:19.771818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.771714 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr4qv\" (UniqueName: \"kubernetes.io/projected/ad066037-9e1a-48be-9db0-f3a9f8768a6b-kube-api-access-rr4qv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.771818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.771748 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/decb8099-9e1b-4e62-8641-9769c2c9c83d-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.771818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.771763 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2f6q\" (UniqueName: \"kubernetes.io/projected/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-kube-api-access-g2f6q\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.771818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.771774 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6lv\" (UniqueName: \"kubernetes.io/projected/6f793472-466a-4b0e-a83f-f21aa9d01c92-kube-api-access-sh6lv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.771818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.771786 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.771818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.771797 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmqk2\" (UniqueName: \"kubernetes.io/projected/decb8099-9e1b-4e62-8641-9769c2c9c83d-kube-api-access-hmqk2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.771818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.771809 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad066037-9e1a-48be-9db0-f3a9f8768a6b-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:19.989519 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.989489 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" path="/var/lib/kubelet/pods/95452683-4ed9-4672-9026-339d1a0a07e5/volumes" Oct 31 22:00:19.989912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:19.989897 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" path="/var/lib/kubelet/pods/e9b968b4-b2d0-4bc7-a8c1-d58735388295/volumes" Oct 31 22:00:20.032092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.031985 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" event={"ID":"86ef7678-8bf6-443c-90e1-8c1af8de24d7","Type":"ContainerDied","Data":"2249a7958410caa9c9ba276edca54f7dccef31cd1a723d4f9ff74b932a5f4a09"} Oct 31 22:00:20.032092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.032063 4294 scope.go:122] "RemoveContainer" containerID="1e5fb394f0c4d2aebbcb0b77d4663e6d59a87f148c8de8b704e7d5c58126c86a" Oct 31 22:00:20.032938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.032909 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j" Oct 31 22:00:20.036150 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.036120 4294 generic.go:356] "Generic (PLEG): container finished" podID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerID="5fc17ba051cb96a865d915e82025b15766d7b304ad347ec687846d8253dda139" exitCode=60 Oct 31 22:00:20.036474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.036222 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerDied","Data":"5fc17ba051cb96a865d915e82025b15766d7b304ad347ec687846d8253dda139"} Oct 31 22:00:20.036718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.036698 4294 scope.go:122] "RemoveContainer" containerID="5fc17ba051cb96a865d915e82025b15766d7b304ad347ec687846d8253dda139" Oct 31 22:00:20.037104 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:20.037075 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv_odh-ai-gateway-infra(46a37b22-93fb-4781-a126-6a138e8ba5c7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" Oct 31 22:00:20.041412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.041386 4294 generic.go:356] "Generic (PLEG): container finished" podID="2237055a-125a-45a4-998f-1ce799325f97" containerID="486c940b5e9420c10251d12d7f73cb8f68be466ace9e5a9f7021013fd2466128" exitCode=60 Oct 31 22:00:20.041585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.041464 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerDied","Data":"486c940b5e9420c10251d12d7f73cb8f68be466ace9e5a9f7021013fd2466128"} Oct 31 22:00:20.041946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.041924 4294 scope.go:122] "RemoveContainer" containerID="486c940b5e9420c10251d12d7f73cb8f68be466ace9e5a9f7021013fd2466128" Oct 31 22:00:20.042532 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:20.042507 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964_odh-ai-gateway-infra(2237055a-125a-45a4-998f-1ce799325f97)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" podUID="2237055a-125a-45a4-998f-1ce799325f97" Oct 31 22:00:20.045233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.045210 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerStarted","Data":"fbf0203aca9ebf7c1dc8517d53516d91989fa2b9a7a59f795b48b93e3624d99c"} Oct 31 22:00:20.051652 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.051632 4294 scope.go:122] "RemoveContainer" containerID="61af786c12e6a24b63bc964877ffa965bea5952bc552aabab167a42dbc8cc7ab" Oct 31 22:00:20.051768 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.051752 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" Oct 31 22:00:20.051862 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.051766 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2" event={"ID":"6e9ff7be-fa18-4a26-83fc-8e6f88ff1713","Type":"ContainerDied","Data":"d4ae3e1643bda5bb491ab23cf25ee9a103c32464e85c04fc15f6d97643ca50f5"} Oct 31 22:00:20.056049 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.056029 4294 generic.go:356] "Generic (PLEG): container finished" podID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerID="50a256976b9e3830ac65ca093622005558b506794f91b77ae1532cc4bfeea15d" exitCode=60 Oct 31 22:00:20.056153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.056101 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerDied","Data":"50a256976b9e3830ac65ca093622005558b506794f91b77ae1532cc4bfeea15d"} Oct 31 22:00:20.056524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.056507 4294 scope.go:122] "RemoveContainer" containerID="50a256976b9e3830ac65ca093622005558b506794f91b77ae1532cc4bfeea15d" Oct 31 22:00:20.056825 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:20.056802 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp_odh-ai-gateway-infra(8994495f-2a50-4a8b-af81-e5c14c57c4fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" Oct 31 22:00:20.058005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.057982 4294 generic.go:356] "Generic (PLEG): container finished" podID="0b742073-6765-4c13-931d-245b538180f6" containerID="0dbf317db7c05eb004fe55b1bfb9c72372ca5fcba502a094241c64b286e9d16b" exitCode=60 Oct 31 22:00:20.058089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.058050 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerDied","Data":"0dbf317db7c05eb004fe55b1bfb9c72372ca5fcba502a094241c64b286e9d16b"} Oct 31 22:00:20.058466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.058448 4294 scope.go:122] "RemoveContainer" containerID="0dbf317db7c05eb004fe55b1bfb9c72372ca5fcba502a094241c64b286e9d16b" Oct 31 22:00:20.058766 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:20.058745 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w_odh-ai-gateway-infra(0b742073-6765-4c13-931d-245b538180f6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" podUID="0b742073-6765-4c13-931d-245b538180f6" Oct 31 22:00:20.059482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.059467 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" Oct 31 22:00:20.059482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.059472 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b" event={"ID":"ad066037-9e1a-48be-9db0-f3a9f8768a6b","Type":"ContainerDied","Data":"f9124f18011b80327b517b080ace591e9bafba03e45aaaf568bd97c36ab13881"} Oct 31 22:00:20.061398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.061377 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" event={"ID":"6f793472-466a-4b0e-a83f-f21aa9d01c92","Type":"ContainerDied","Data":"2f32f680a70977ab02faabe862e32c85d24637545c5655fe2f90e0787f08728e"} Oct 31 22:00:20.061564 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.061388 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4" Oct 31 22:00:20.063187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.063158 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" event={"ID":"decb8099-9e1b-4e62-8641-9769c2c9c83d","Type":"ContainerDied","Data":"fac0724f9c3c33bd5f422229712eb33038a125f235120ecccbcefd889bab7444"} Oct 31 22:00:20.063271 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.063192 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5" Oct 31 22:00:20.068471 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.068451 4294 scope.go:122] "RemoveContainer" containerID="8d4520779f8c1c3ea61e775146838e77d40a540b243beb9f3a980e18211c2714" Oct 31 22:00:20.086929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.086907 4294 scope.go:122] "RemoveContainer" containerID="de478d8a1722ba6fd1888837b88bbb824684b0e48851b0b19f9aace577e68a15" Oct 31 22:00:20.157216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.157179 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w"] Oct 31 22:00:20.168146 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.167787 4294 scope.go:122] "RemoveContainer" containerID="26484b008efef826e1f9f89c1e0c101f9c86107824e8cbf99f31f389e17e4b01" Oct 31 22:00:20.184013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.183994 4294 scope.go:122] "RemoveContainer" containerID="5109bb6c8e15394b01d8d3c81d51390758f60166b611881c5927e1feb2e41486" Oct 31 22:00:20.200348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.200322 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp"] Oct 31 22:00:20.224094 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.224065 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j"] Oct 31 22:00:20.232089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.232060 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4275b9-29891400-kgh7j"] Oct 31 22:00:20.232386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.232368 4294 scope.go:122] "RemoveContainer" containerID="3610120a95a11749cc29019a3fcbab19a22a62779081ae7a50640b0b53efb8a6" Oct 31 22:00:20.234445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.234427 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:20.242076 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.242048 4294 scope.go:122] "RemoveContainer" containerID="bdf10a3fca70b2c7371b67e61f1ad81936d86514496642b4ab0d91ffc8b50e91" Oct 31 22:00:20.250857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.250838 4294 scope.go:122] "RemoveContainer" containerID="afb0f6c1a1617a4bfaf977b49e4b0078aa905cc4849b57af55830d3aff5a7258" Oct 31 22:00:20.265203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.265181 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5"] Oct 31 22:00:20.275135 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.275095 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891400-mdbq5"] Oct 31 22:00:20.292913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.292842 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964"] Oct 31 22:00:20.315084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.315056 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4"] Oct 31 22:00:20.318682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.318654 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891400-czfp4"] Oct 31 22:00:20.356350 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.356322 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b"] Oct 31 22:00:20.359826 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.359799 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-696f9a-29891400-mht5b"] Oct 31 22:00:20.381915 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.381891 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-kube-api-access-js4mj\" (UniqueName: \"kubernetes.io/projected/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-kube-api-access-js4mj\") pod \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " Oct 31 22:00:20.382035 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.382026 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-maas-api-service-ca\") pod \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\" (UID: \"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd\") " Oct 31 22:00:20.382428 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.382400 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-maas-api-service-ca" pod "ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" (UID: "ec3be5fe-e715-4121-bd7b-e76b9fb6edbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:20.382604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.382586 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:20.384012 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.383987 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-kube-api-access-js4mj" pod "ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" (UID: "ec3be5fe-e715-4121-bd7b-e76b9fb6edbd"). InnerVolumeSpecName "kube-api-access-js4mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:20.391899 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.391879 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv"] Oct 31 22:00:20.484896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.484852 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4mj\" (UniqueName: \"kubernetes.io/projected/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd-kube-api-access-js4mj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:20.488377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.488335 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2"] Oct 31 22:00:20.494707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:20.494681 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4b5989-29891400-ppbg2"] Oct 31 22:00:21.077018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.076988 4294 generic.go:356] "Generic (PLEG): container finished" podID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerID="fbf0203aca9ebf7c1dc8517d53516d91989fa2b9a7a59f795b48b93e3624d99c" exitCode=60 Oct 31 22:00:21.077493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.077057 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerDied","Data":"fbf0203aca9ebf7c1dc8517d53516d91989fa2b9a7a59f795b48b93e3624d99c"} Oct 31 22:00:21.077493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.077125 4294 scope.go:122] "RemoveContainer" containerID="cedbba0ab772cc4ce3bac62a01aa5a628764c8499ce26a2dedbf382449827600" Oct 31 22:00:21.077599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.077557 4294 scope.go:122] "RemoveContainer" containerID="fbf0203aca9ebf7c1dc8517d53516d91989fa2b9a7a59f795b48b93e3624d99c" Oct 31 22:00:21.077963 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:21.077943 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb_odh-ai-gateway-infra(0115189e-cf8e-4af0-b4d6-da4d82bd9db0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" Oct 31 22:00:21.079520 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.079500 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" Oct 31 22:00:21.079629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.079507 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h" event={"ID":"ec3be5fe-e715-4121-bd7b-e76b9fb6edbd","Type":"ContainerDied","Data":"e439fa371a90d5ed838df6fc9ba89dcfa03b6fe5548501f0e8fa2176ac51b7d7"} Oct 31 22:00:21.119350 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.119318 4294 scope.go:122] "RemoveContainer" containerID="784112f6c0e5415923c808622a6ee967b2e09f32020063d85cb63b4d141ec93b" Oct 31 22:00:21.169741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.150957 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb"] Oct 31 22:00:21.319718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.319688 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h"] Oct 31 22:00:21.326094 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.326030 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891400-glh6h"] Oct 31 22:00:21.332984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.332965 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:21.363595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.363570 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:21.368851 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.368824 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:21.373366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.373346 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:21.418774 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.418748 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2237055a-125a-45a4-998f-1ce799325f97-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2237055a-125a-45a4-998f-1ce799325f97-maas-api-service-ca\") pod \"2237055a-125a-45a4-998f-1ce799325f97\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " Oct 31 22:00:21.418924 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.418892 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2237055a-125a-45a4-998f-1ce799325f97-kube-api-access-gch4b\" (UniqueName: \"kubernetes.io/projected/2237055a-125a-45a4-998f-1ce799325f97-kube-api-access-gch4b\") pod \"2237055a-125a-45a4-998f-1ce799325f97\" (UID: \"2237055a-125a-45a4-998f-1ce799325f97\") " Oct 31 22:00:21.419150 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.419129 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2237055a-125a-45a4-998f-1ce799325f97-maas-api-service-ca" pod "2237055a-125a-45a4-998f-1ce799325f97" (UID: "2237055a-125a-45a4-998f-1ce799325f97"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:21.419522 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.419492 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2237055a-125a-45a4-998f-1ce799325f97-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.421490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.421465 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2237055a-125a-45a4-998f-1ce799325f97-kube-api-access-gch4b" pod "2237055a-125a-45a4-998f-1ce799325f97" (UID: "2237055a-125a-45a4-998f-1ce799325f97"). InnerVolumeSpecName "kube-api-access-gch4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:21.520148 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520109 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b742073-6765-4c13-931d-245b538180f6-kube-api-access-ctk9z\" (UniqueName: \"kubernetes.io/projected/0b742073-6765-4c13-931d-245b538180f6-kube-api-access-ctk9z\") pod \"0b742073-6765-4c13-931d-245b538180f6\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " Oct 31 22:00:21.520362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520250 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46a37b22-93fb-4781-a126-6a138e8ba5c7-kube-api-access-qtd4s\" (UniqueName: \"kubernetes.io/projected/46a37b22-93fb-4781-a126-6a138e8ba5c7-kube-api-access-qtd4s\") pod \"46a37b22-93fb-4781-a126-6a138e8ba5c7\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " Oct 31 22:00:21.520431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520356 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b742073-6765-4c13-931d-245b538180f6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b742073-6765-4c13-931d-245b538180f6-maas-api-service-ca\") pod \"0b742073-6765-4c13-931d-245b538180f6\" (UID: \"0b742073-6765-4c13-931d-245b538180f6\") " Oct 31 22:00:21.520431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520404 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8994495f-2a50-4a8b-af81-e5c14c57c4fc-kube-api-access-lkf65\" (UniqueName: \"kubernetes.io/projected/8994495f-2a50-4a8b-af81-e5c14c57c4fc-kube-api-access-lkf65\") pod \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " Oct 31 22:00:21.520554 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520540 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46a37b22-93fb-4781-a126-6a138e8ba5c7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a37b22-93fb-4781-a126-6a138e8ba5c7-maas-api-service-ca\") pod \"46a37b22-93fb-4781-a126-6a138e8ba5c7\" (UID: \"46a37b22-93fb-4781-a126-6a138e8ba5c7\") " Oct 31 22:00:21.520686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520670 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8994495f-2a50-4a8b-af81-e5c14c57c4fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8994495f-2a50-4a8b-af81-e5c14c57c4fc-maas-api-service-ca\") pod \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\" (UID: \"8994495f-2a50-4a8b-af81-e5c14c57c4fc\") " Oct 31 22:00:21.521339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520726 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b742073-6765-4c13-931d-245b538180f6-maas-api-service-ca" pod "0b742073-6765-4c13-931d-245b538180f6" (UID: "0b742073-6765-4c13-931d-245b538180f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:21.521339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.520895 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a37b22-93fb-4781-a126-6a138e8ba5c7-maas-api-service-ca" pod "46a37b22-93fb-4781-a126-6a138e8ba5c7" (UID: "46a37b22-93fb-4781-a126-6a138e8ba5c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:21.521339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.521042 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8994495f-2a50-4a8b-af81-e5c14c57c4fc-maas-api-service-ca" pod "8994495f-2a50-4a8b-af81-e5c14c57c4fc" (UID: "8994495f-2a50-4a8b-af81-e5c14c57c4fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:21.521533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.521357 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8994495f-2a50-4a8b-af81-e5c14c57c4fc-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.521533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.521378 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gch4b\" (UniqueName: \"kubernetes.io/projected/2237055a-125a-45a4-998f-1ce799325f97-kube-api-access-gch4b\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.521533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.521391 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b742073-6765-4c13-931d-245b538180f6-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.521533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.521405 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a37b22-93fb-4781-a126-6a138e8ba5c7-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.522991 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.522964 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b742073-6765-4c13-931d-245b538180f6-kube-api-access-ctk9z" pod "0b742073-6765-4c13-931d-245b538180f6" (UID: "0b742073-6765-4c13-931d-245b538180f6"). InnerVolumeSpecName "kube-api-access-ctk9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:21.523079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.522996 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a37b22-93fb-4781-a126-6a138e8ba5c7-kube-api-access-qtd4s" pod "46a37b22-93fb-4781-a126-6a138e8ba5c7" (UID: "46a37b22-93fb-4781-a126-6a138e8ba5c7"). InnerVolumeSpecName "kube-api-access-qtd4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:21.523079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.523021 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8994495f-2a50-4a8b-af81-e5c14c57c4fc-kube-api-access-lkf65" pod "8994495f-2a50-4a8b-af81-e5c14c57c4fc" (UID: "8994495f-2a50-4a8b-af81-e5c14c57c4fc"). InnerVolumeSpecName "kube-api-access-lkf65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:21.622444 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:21.622339 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-192c5e9d9ff4: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:21.622444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.622351 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctk9z\" (UniqueName: \"kubernetes.io/projected/0b742073-6765-4c13-931d-245b538180f6-kube-api-access-ctk9z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.622444 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:21.622370 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:21.622444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.622379 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtd4s\" (UniqueName: \"kubernetes.io/projected/46a37b22-93fb-4781-a126-6a138e8ba5c7-kube-api-access-qtd4s\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.622444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.622392 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkf65\" (UniqueName: \"kubernetes.io/projected/8994495f-2a50-4a8b-af81-e5c14c57c4fc-kube-api-access-lkf65\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:21.622444 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:21.622422 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials podName:c977fc93-3845-44c2-a8b8-3bdafbcd2656 nodeName:}" failed. No retries permitted until 2026-10-31 22:00:25.622409539 +0000 UTC m=+2665.747516752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656") : secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:21.990100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.990062 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" path="/var/lib/kubelet/pods/6e9ff7be-fa18-4a26-83fc-8e6f88ff1713/volumes" Oct 31 22:00:21.990764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.990744 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" path="/var/lib/kubelet/pods/6f793472-466a-4b0e-a83f-f21aa9d01c92/volumes" Oct 31 22:00:21.991178 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.991165 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" path="/var/lib/kubelet/pods/86ef7678-8bf6-443c-90e1-8c1af8de24d7/volumes" Oct 31 22:00:21.991890 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.991875 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" path="/var/lib/kubelet/pods/ad066037-9e1a-48be-9db0-f3a9f8768a6b/volumes" Oct 31 22:00:21.992260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.992248 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" path="/var/lib/kubelet/pods/decb8099-9e1b-4e62-8641-9769c2c9c83d/volumes" Oct 31 22:00:21.992645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:21.992633 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" path="/var/lib/kubelet/pods/ec3be5fe-e715-4121-bd7b-e76b9fb6edbd/volumes" Oct 31 22:00:22.099218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.099188 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" Oct 31 22:00:22.099687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.099194 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp" event={"ID":"8994495f-2a50-4a8b-af81-e5c14c57c4fc","Type":"ContainerDied","Data":"383a65ab8cf3272850c8d97251986c6b1f9959339a3e2ecb1149191768b1c453"} Oct 31 22:00:22.099687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.099339 4294 scope.go:122] "RemoveContainer" containerID="50a256976b9e3830ac65ca093622005558b506794f91b77ae1532cc4bfeea15d" Oct 31 22:00:22.100773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.100750 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" Oct 31 22:00:22.100866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.100773 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w" event={"ID":"0b742073-6765-4c13-931d-245b538180f6","Type":"ContainerDied","Data":"22196fc4896d654decb92097053b6eeb9075ecbb4b1e8d24f1c30551c4d1a396"} Oct 31 22:00:22.102693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.102675 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" Oct 31 22:00:22.102806 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.102672 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv" event={"ID":"46a37b22-93fb-4781-a126-6a138e8ba5c7","Type":"ContainerDied","Data":"64e07d927082cb82fdd7adb5b88453bc64c5334cf4f63438b0916fca35e85323"} Oct 31 22:00:22.104447 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.104429 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" Oct 31 22:00:22.104447 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.104434 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964" event={"ID":"2237055a-125a-45a4-998f-1ce799325f97","Type":"ContainerDied","Data":"2cfde700f8f2688117b73684c6880a8c6a0a7ea9a17d1f6d04d8b5d69fe592bb"} Oct 31 22:00:22.114711 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.114691 4294 scope.go:122] "RemoveContainer" containerID="0dbf317db7c05eb004fe55b1bfb9c72372ca5fcba502a094241c64b286e9d16b" Oct 31 22:00:22.124685 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.124666 4294 scope.go:122] "RemoveContainer" containerID="5fc17ba051cb96a865d915e82025b15766d7b304ad347ec687846d8253dda139" Oct 31 22:00:22.134640 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.134618 4294 scope.go:122] "RemoveContainer" containerID="486c940b5e9420c10251d12d7f73cb8f68be466ace9e5a9f7021013fd2466128" Oct 31 22:00:22.155742 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.155718 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w"] Oct 31 22:00:22.163185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.163156 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891400-7m88w"] Oct 31 22:00:22.222485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.222447 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp"] Oct 31 22:00:22.225411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.225382 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-994336-29891400-9hrxp"] Oct 31 22:00:22.267001 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.257296 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964"] Oct 31 22:00:22.267001 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.261457 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-943963-29891400-z9964"] Oct 31 22:00:22.283470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.282186 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:22.309677 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.308373 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv"] Oct 31 22:00:22.314402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.314368 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891400-qrpmv"] Oct 31 22:00:22.465828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.440912 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-maas-api-service-ca\") pod \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " Oct 31 22:00:22.481890 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.474675 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-kube-api-access-fz5hj\" (UniqueName: \"kubernetes.io/projected/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-kube-api-access-fz5hj\") pod \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\" (UID: \"0115189e-cf8e-4af0-b4d6-da4d82bd9db0\") " Oct 31 22:00:22.488338 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.482815 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-maas-api-service-ca" pod "0115189e-cf8e-4af0-b4d6-da4d82bd9db0" (UID: "0115189e-cf8e-4af0-b4d6-da4d82bd9db0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:22.507005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.506928 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-kube-api-access-fz5hj" pod "0115189e-cf8e-4af0-b4d6-da4d82bd9db0" (UID: "0115189e-cf8e-4af0-b4d6-da4d82bd9db0"). InnerVolumeSpecName "kube-api-access-fz5hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:22.511381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.508660 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:22.613537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.613504 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz5hj\" (UniqueName: \"kubernetes.io/projected/0115189e-cf8e-4af0-b4d6-da4d82bd9db0-kube-api-access-fz5hj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:22.965553 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.965465 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z"] Oct 31 22:00:22.966110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966096 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:22.966154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966112 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:22.966154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966125 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:22.966154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966130 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:22.966154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966143 4294 cpu_manager.go:414] "containerMap: removing container" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:22.966154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966148 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:22.966154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966155 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966160 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966172 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966176 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966183 4294 cpu_manager.go:414] "containerMap: removing container" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966187 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966193 4294 cpu_manager.go:414] "containerMap: removing container" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966197 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966203 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966206 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966213 4294 cpu_manager.go:414] "containerMap: removing container" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966217 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966226 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966230 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966236 4294 cpu_manager.go:414] "containerMap: removing container" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966240 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966246 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966251 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966257 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966261 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966270 4294 cpu_manager.go:414] "containerMap: removing container" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966275 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966281 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966286 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966292 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966297 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966316 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0a44c138-1a2c-4bb9-b57c-32155db4050e" containerName="revoke-keys" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966321 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a44c138-1a2c-4bb9-b57c-32155db4050e" containerName="revoke-keys" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966329 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966333 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966339 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966344 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966353 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966357 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966366 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966370 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966383 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966387 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966394 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966398 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966405 4294 cpu_manager.go:414] "containerMap: removing container" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966409 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966416 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966421 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.966403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966427 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966431 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966439 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966443 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966451 4294 cpu_manager.go:414] "containerMap: removing container" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966456 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966462 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966466 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966473 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966477 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966568 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966577 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966584 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966589 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966594 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966601 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966605 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966611 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966616 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966623 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966628 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966633 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966637 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966642 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966649 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966653 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a44c138-1a2c-4bb9-b57c-32155db4050e" containerName="revoke-keys" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966659 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966664 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966669 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966675 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966680 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966685 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966690 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966695 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966700 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966704 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966710 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966715 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966720 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966725 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:22.967762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.966730 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f793472-466a-4b0e-a83f-f21aa9d01c92" containerName="cleanup" Oct 31 22:00:22.970398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:22.970382 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.036377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.025609 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:00:23.036377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.028542 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvbks\" (UniqueName: \"kubernetes.io/projected/e808011d-7e69-4029-b11f-d22c647262e3-kube-api-access-dvbks\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.036377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.028660 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.036377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.028713 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.036377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.032089 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z"] Oct 31 22:00:23.130349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.130292 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvbks\" (UniqueName: \"kubernetes.io/projected/e808011d-7e69-4029-b11f-d22c647262e3-kube-api-access-dvbks\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.131020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.130994 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.131224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.131205 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.132451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.132429 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.139029 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.137614 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.175548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.175483 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvbks\" (UniqueName: \"kubernetes.io/projected/e808011d-7e69-4029-b11f-d22c647262e3-kube-api-access-dvbks\") pod \"maas-api-revoke-sub-a7e2967b-kxw2z\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.179146 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.179106 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" event={"ID":"0115189e-cf8e-4af0-b4d6-da4d82bd9db0","Type":"ContainerDied","Data":"9b4154db53dffbdc223d1179ff53585c27982362cb73f32508417e50a1c4c61b"} Oct 31 22:00:23.179290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.179165 4294 scope.go:122] "RemoveContainer" containerID="fbf0203aca9ebf7c1dc8517d53516d91989fa2b9a7a59f795b48b93e3624d99c" Oct 31 22:00:23.179290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.179276 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb" Oct 31 22:00:23.234858 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.234833 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb"] Oct 31 22:00:23.239261 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.239239 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-0877f5-29891400-mbqgb"] Oct 31 22:00:23.369907 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.369869 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:23.676039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.676008 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z"] Oct 31 22:00:23.676154 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:23.676057 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode808011d_7e69_4029_b11f_d22c647262e3.slice/crio-7e88a7f59b8977c4f2e056aeaafc7272d3164063d2b1d459dd0a228425793426 WatchSource:0}: Error finding container 7e88a7f59b8977c4f2e056aeaafc7272d3164063d2b1d459dd0a228425793426: Status 404 returned error can't find the container with id 7e88a7f59b8977c4f2e056aeaafc7272d3164063d2b1d459dd0a228425793426 Oct 31 22:00:23.985781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.985551 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" path="/var/lib/kubelet/pods/0115189e-cf8e-4af0-b4d6-da4d82bd9db0/volumes" Oct 31 22:00:23.986159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.986136 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b742073-6765-4c13-931d-245b538180f6" path="/var/lib/kubelet/pods/0b742073-6765-4c13-931d-245b538180f6/volumes" Oct 31 22:00:23.986577 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.986559 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2237055a-125a-45a4-998f-1ce799325f97" path="/var/lib/kubelet/pods/2237055a-125a-45a4-998f-1ce799325f97/volumes" Oct 31 22:00:23.987485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.987466 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" path="/var/lib/kubelet/pods/46a37b22-93fb-4781-a126-6a138e8ba5c7/volumes" Oct 31 22:00:23.988020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:23.988004 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8994495f-2a50-4a8b-af81-e5c14c57c4fc" path="/var/lib/kubelet/pods/8994495f-2a50-4a8b-af81-e5c14c57c4fc/volumes" Oct 31 22:00:24.194039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:24.194004 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" event={"ID":"e808011d-7e69-4029-b11f-d22c647262e3","Type":"ContainerStarted","Data":"a5077684715094f6d424da79b7fc18e959c6837abcb5d1a6c2bbf676fdcd60e0"} Oct 31 22:00:24.194549 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:24.194069 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" event={"ID":"e808011d-7e69-4029-b11f-d22c647262e3","Type":"ContainerStarted","Data":"7e88a7f59b8977c4f2e056aeaafc7272d3164063d2b1d459dd0a228425793426"} Oct 31 22:00:24.236401 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:24.236273 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" podStartSLOduration=2.236260369 podStartE2EDuration="2.236260369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:24.234222051 +0000 UTC m=+2664.359329282" watchObservedRunningTime="2026-10-31 22:00:24.236260369 +0000 UTC m=+2664.361367598" Oct 31 22:00:25.214532 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:25.214497 4294 generic.go:356] "Generic (PLEG): container finished" podID="e808011d-7e69-4029-b11f-d22c647262e3" containerID="a5077684715094f6d424da79b7fc18e959c6837abcb5d1a6c2bbf676fdcd60e0" exitCode=0 Oct 31 22:00:25.214906 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:25.214584 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" event={"ID":"e808011d-7e69-4029-b11f-d22c647262e3","Type":"ContainerDied","Data":"a5077684715094f6d424da79b7fc18e959c6837abcb5d1a6c2bbf676fdcd60e0"} Oct 31 22:00:25.666861 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:25.666828 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-192c5e9d9ff4: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:25.666861 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:25.666855 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-da7a4611d93a: secret "agc-cred-da7a4611d93a" not found Oct 31 22:00:25.666861 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:25.666870 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx: [secret "agc-cred-192c5e9d9ff4" not found, secret "agc-cred-da7a4611d93a" not found] Oct 31 22:00:25.667129 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:25.666920 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials podName:c977fc93-3845-44c2-a8b8-3bdafbcd2656 nodeName:}" failed. No retries permitted until 2026-10-31 22:00:33.666907692 +0000 UTC m=+2673.792014914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656") : [secret "agc-cred-192c5e9d9ff4" not found, secret "agc-cred-da7a4611d93a" not found] Oct 31 22:00:26.375547 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.375522 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:26.477629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.477547 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-xks-ca\") pod \"e808011d-7e69-4029-b11f-d22c647262e3\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " Oct 31 22:00:26.477847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.477771 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e808011d-7e69-4029-b11f-d22c647262e3-kube-api-access-dvbks\" (UniqueName: \"kubernetes.io/projected/e808011d-7e69-4029-b11f-d22c647262e3-kube-api-access-dvbks\") pod \"e808011d-7e69-4029-b11f-d22c647262e3\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " Oct 31 22:00:26.477847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.477835 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-openshift-ca\") pod \"e808011d-7e69-4029-b11f-d22c647262e3\" (UID: \"e808011d-7e69-4029-b11f-d22c647262e3\") " Oct 31 22:00:26.478245 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.478220 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-openshift-ca" pod "e808011d-7e69-4029-b11f-d22c647262e3" (UID: "e808011d-7e69-4029-b11f-d22c647262e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:26.478394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.478376 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:26.479546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.479523 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-xks-ca" pod "e808011d-7e69-4029-b11f-d22c647262e3" (UID: "e808011d-7e69-4029-b11f-d22c647262e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:00:26.479735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.479713 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e808011d-7e69-4029-b11f-d22c647262e3-kube-api-access-dvbks" pod "e808011d-7e69-4029-b11f-d22c647262e3" (UID: "e808011d-7e69-4029-b11f-d22c647262e3"). InnerVolumeSpecName "kube-api-access-dvbks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:26.579545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.579516 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvbks\" (UniqueName: \"kubernetes.io/projected/e808011d-7e69-4029-b11f-d22c647262e3-kube-api-access-dvbks\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:26.579545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:26.579543 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e808011d-7e69-4029-b11f-d22c647262e3-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:27.251263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:27.251199 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" Oct 31 22:00:27.251469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:27.251281 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z" event={"ID":"e808011d-7e69-4029-b11f-d22c647262e3","Type":"ContainerDied","Data":"7e88a7f59b8977c4f2e056aeaafc7272d3164063d2b1d459dd0a228425793426"} Oct 31 22:00:27.251469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:27.251359 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e88a7f59b8977c4f2e056aeaafc7272d3164063d2b1d459dd0a228425793426" Oct 31 22:00:31.333277 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:31.333236 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:00:33.746691 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:33.746657 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-192c5e9d9ff4: secret "agc-cred-192c5e9d9ff4" not found Oct 31 22:00:33.746691 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:33.746688 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4a73b1/agc-cred-da7a4611d93a: secret "agc-cred-da7a4611d93a" not found Oct 31 22:00:33.746691 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:33.746701 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx: [secret "agc-cred-192c5e9d9ff4" not found, secret "agc-cred-da7a4611d93a" not found] Oct 31 22:00:33.747503 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:33.746763 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials podName:c977fc93-3845-44c2-a8b8-3bdafbcd2656 nodeName:}" failed. No retries permitted until 2026-10-31 22:00:49.746748937 +0000 UTC m=+2689.871856159 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656") : [secret "agc-cred-192c5e9d9ff4" not found, secret "agc-cred-da7a4611d93a" not found] Oct 31 22:00:34.255451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:34.255399 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48"] Oct 31 22:00:34.255706 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:34.255670 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" containerID="cri-o://a302136dccbfca3fc586f3331234449f8e7db4e6a4f669226a2ad2b4f5fa5ac4" gracePeriod=30 Oct 31 22:00:34.325577 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:34.325536 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx"] Oct 31 22:00:37.897467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:37.897432 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:00:37.897860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:37.897502 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:00:39.327123 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.327078 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" podUID="c977fc93-3845-44c2-a8b8-3bdafbcd2656" containerName="payload-processing" containerID="cri-o://872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc" gracePeriod=25 Oct 31 22:00:39.419384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.419350 4294 generic.go:356] "Generic (PLEG): container finished" podID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerID="a302136dccbfca3fc586f3331234449f8e7db4e6a4f669226a2ad2b4f5fa5ac4" exitCode=0 Oct 31 22:00:39.419545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.419425 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" event={"ID":"5eae5971-48ba-41ff-bc32-4580174bd6c2","Type":"ContainerDied","Data":"a302136dccbfca3fc586f3331234449f8e7db4e6a4f669226a2ad2b4f5fa5ac4"} Oct 31 22:00:39.421097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.421076 4294 generic.go:356] "Generic (PLEG): container finished" podID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerID="95d235eab51805239e7b995f5eeb1c7692cbc39ebdd6ba2089788d2411b31379" exitCode=0 Oct 31 22:00:39.421228 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.421150 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" event={"ID":"a59a4817-e90a-4639-8abe-7f1584fa5eb6","Type":"ContainerDied","Data":"95d235eab51805239e7b995f5eeb1c7692cbc39ebdd6ba2089788d2411b31379"} Oct 31 22:00:39.421624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.421610 4294 scope.go:122] "RemoveContainer" containerID="95d235eab51805239e7b995f5eeb1c7692cbc39ebdd6ba2089788d2411b31379" Oct 31 22:00:39.423352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.423327 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:00:39.423437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.423390 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:00:39.423437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.423425 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:00:39.682758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.682678 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 22:00:39.716626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.716596 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 22:00:39.788406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788375 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-kube-api-access-zv84n\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-kube-api-access-zv84n\") pod \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " Oct 31 22:00:39.788597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788413 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-podinfo\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.788597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788442 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-data\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.788597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788469 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-kube-api-access-t5zcs\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-kube-api-access-t5zcs\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.788597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788529 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-certs\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.788597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788566 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-plugins-config-volume\") pod \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " Oct 31 22:00:39.788886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788626 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-envoy\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.788886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788660 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eae5971-48ba-41ff-bc32-4580174bd6c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae5971-48ba-41ff-bc32-4580174bd6c2-istiod-ca-cert\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.788886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788774 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials\") pod \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " Oct 31 22:00:39.788886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788846 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-routing-overlay-volume\") pod \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\" (UID: \"c977fc93-3845-44c2-a8b8-3bdafbcd2656\") " Oct 31 22:00:39.789107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.788943 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-credential-socket\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.789107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.789020 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-socket\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.789229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.789119 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-token\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-token\") pod \"5eae5971-48ba-41ff-bc32-4580174bd6c2\" (UID: \"5eae5971-48ba-41ff-bc32-4580174bd6c2\") " Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.789595 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-certs" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.789829 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.790055 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-credential-socket" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.790405 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-routing-overlay-volume" pod "c977fc93-3845-44c2-a8b8-3bdafbcd2656" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.790700 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-socket" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.791062 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eae5971-48ba-41ff-bc32-4580174bd6c2-istiod-ca-cert" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.791520 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-plugins-config-volume" pod "c977fc93-3845-44c2-a8b8-3bdafbcd2656" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.791539 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials" pod "c977fc93-3845-44c2-a8b8-3bdafbcd2656" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:39.791614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.791586 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-data" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.792128 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.791992 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-token" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:39.792646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.792624 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-kube-api-access-zv84n" pod "c977fc93-3845-44c2-a8b8-3bdafbcd2656" (UID: "c977fc93-3845-44c2-a8b8-3bdafbcd2656"). InnerVolumeSpecName "kube-api-access-zv84n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:39.794362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.794327 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-podinfo" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:00:39.794749 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.794718 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-envoy" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:00:39.795783 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.795736 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-kube-api-access-t5zcs" pod "5eae5971-48ba-41ff-bc32-4580174bd6c2" (UID: "5eae5971-48ba-41ff-bc32-4580174bd6c2"). InnerVolumeSpecName "kube-api-access-t5zcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890874 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890901 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890913 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890922 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890930 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv84n\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-kube-api-access-zv84n\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890938 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890947 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.890948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890956 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5zcs\" (UniqueName: \"kubernetes.io/projected/5eae5971-48ba-41ff-bc32-4580174bd6c2-kube-api-access-t5zcs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.891368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890964 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c977fc93-3845-44c2-a8b8-3bdafbcd2656-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.891368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890974 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eae5971-48ba-41ff-bc32-4580174bd6c2-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.891368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890981 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eae5971-48ba-41ff-bc32-4580174bd6c2-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:39.891368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:39.890989 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c977fc93-3845-44c2-a8b8-3bdafbcd2656-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:40.431232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.431192 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" event={"ID":"a59a4817-e90a-4639-8abe-7f1584fa5eb6","Type":"ContainerStarted","Data":"22961a112a97cc771ff397f9944dc2e70f643d9737d85f6f1e4ae4cce404c2e5"} Oct 31 22:00:40.432729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.432697 4294 generic.go:356] "Generic (PLEG): container finished" podID="c977fc93-3845-44c2-a8b8-3bdafbcd2656" containerID="872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc" exitCode=0 Oct 31 22:00:40.432867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.432756 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" Oct 31 22:00:40.432867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.432778 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" event={"ID":"c977fc93-3845-44c2-a8b8-3bdafbcd2656","Type":"ContainerDied","Data":"872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc"} Oct 31 22:00:40.432867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.432825 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx" event={"ID":"c977fc93-3845-44c2-a8b8-3bdafbcd2656","Type":"ContainerDied","Data":"395959dc19ed6faa0ac46e0d06de39153da7dfd63868ce330328009c073ee568"} Oct 31 22:00:40.432867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.432847 4294 scope.go:122] "RemoveContainer" containerID="872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc" Oct 31 22:00:40.434542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.434517 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" event={"ID":"5eae5971-48ba-41ff-bc32-4580174bd6c2","Type":"ContainerDied","Data":"d83009e7bcd63e0026600313a069d8ebf45ff773d5a5cc73bd3dec68f73733ae"} Oct 31 22:00:40.434636 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.434552 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48" Oct 31 22:00:40.443443 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.443424 4294 scope.go:122] "RemoveContainer" containerID="872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc" Oct 31 22:00:40.443737 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:40.443688 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc\": container with ID starting with 872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc not found: ID does not exist" containerID="872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc" Oct 31 22:00:40.443737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.443719 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc"} err="failed to get container status \"872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc\": rpc error: code = NotFound desc = could not find container \"872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc\": container with ID starting with 872b381238230f13f42543cddfe4507ef9d5a86d7c50d07e4f57bcfa096333dc not found: ID does not exist" Oct 31 22:00:40.443870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.443740 4294 scope.go:122] "RemoveContainer" containerID="a302136dccbfca3fc586f3331234449f8e7db4e6a4f669226a2ad2b4f5fa5ac4" Oct 31 22:00:40.535285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.535251 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48"] Oct 31 22:00:40.541330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.541292 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-gxf48"] Oct 31 22:00:40.600422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.600358 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx"] Oct 31 22:00:40.603241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:40.603213 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-675bccjpx"] Oct 31 22:00:41.161464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:41.161430 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 22:00:41.161651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:41.161583 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 22:00:41.161697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:41.161647 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 22:00:42.004588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:42.004556 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" path="/var/lib/kubelet/pods/5eae5971-48ba-41ff-bc32-4580174bd6c2/volumes" Oct 31 22:00:42.005095 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:42.005078 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c977fc93-3845-44c2-a8b8-3bdafbcd2656" path="/var/lib/kubelet/pods/c977fc93-3845-44c2-a8b8-3bdafbcd2656/volumes" Oct 31 22:00:42.161329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:42.161285 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 22:00:42.161498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:42.161353 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 22:00:43.161279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:43.161241 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 22:00:43.161702 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:43.161314 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 22:00:44.161974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:44.161940 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 22:00:44.162400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:44.161998 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 22:00:45.161865 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:45.161826 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 31 22:00:45.162071 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:45.161891 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 31 22:00:45.204227 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:45.204193 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh"] Oct 31 22:00:46.166823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:46.166792 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 22:00:46.167330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:46.167294 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 22:00:46.167993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:46.167974 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 22:00:50.204778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.204706 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" podUID="62c78436-f731-4d09-9d46-cfa4540c6b08" containerName="payload-processing" containerID="cri-o://c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973" gracePeriod=25 Oct 31 22:00:50.464430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.464372 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 22:00:50.516562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.516525 4294 generic.go:356] "Generic (PLEG): container finished" podID="62c78436-f731-4d09-9d46-cfa4540c6b08" containerID="c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973" exitCode=0 Oct 31 22:00:50.516729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.516575 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" Oct 31 22:00:50.516729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.516597 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" event={"ID":"62c78436-f731-4d09-9d46-cfa4540c6b08","Type":"ContainerDied","Data":"c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973"} Oct 31 22:00:50.516729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.516668 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh" event={"ID":"62c78436-f731-4d09-9d46-cfa4540c6b08","Type":"ContainerDied","Data":"f94dd42face3d479174b9e0b16346fa4661f06482d7675651d054f6cf2a7fa27"} Oct 31 22:00:50.516729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.516700 4294 scope.go:122] "RemoveContainer" containerID="c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973" Oct 31 22:00:50.529240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.529220 4294 scope.go:122] "RemoveContainer" containerID="c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973" Oct 31 22:00:50.529746 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:00:50.529723 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973\": container with ID starting with c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973 not found: ID does not exist" containerID="c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973" Oct 31 22:00:50.529821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.529755 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973"} err="failed to get container status \"c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973\": rpc error: code = NotFound desc = could not find container \"c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973\": container with ID starting with c44de83eef83f02ed087dc80e58062b55f98a8dd640a5e98adf63a40ed792973 not found: ID does not exist" Oct 31 22:00:50.599713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.599689 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-kube-api-access-w2j7v\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-kube-api-access-w2j7v\") pod \"62c78436-f731-4d09-9d46-cfa4540c6b08\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " Oct 31 22:00:50.599884 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.599818 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-routing-overlay-volume\") pod \"62c78436-f731-4d09-9d46-cfa4540c6b08\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " Oct 31 22:00:50.599884 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.599869 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-provider-credentials\") pod \"62c78436-f731-4d09-9d46-cfa4540c6b08\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " Oct 31 22:00:50.600005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.599963 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-plugins-config-volume\") pod \"62c78436-f731-4d09-9d46-cfa4540c6b08\" (UID: \"62c78436-f731-4d09-9d46-cfa4540c6b08\") " Oct 31 22:00:50.600248 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.600160 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-routing-overlay-volume" pod "62c78436-f731-4d09-9d46-cfa4540c6b08" (UID: "62c78436-f731-4d09-9d46-cfa4540c6b08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:50.600418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.600396 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-plugins-config-volume" pod "62c78436-f731-4d09-9d46-cfa4540c6b08" (UID: "62c78436-f731-4d09-9d46-cfa4540c6b08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:00:50.600642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.600626 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:50.600684 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.600645 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62c78436-f731-4d09-9d46-cfa4540c6b08-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:50.601844 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.601827 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-kube-api-access-w2j7v" pod "62c78436-f731-4d09-9d46-cfa4540c6b08" (UID: "62c78436-f731-4d09-9d46-cfa4540c6b08"). InnerVolumeSpecName "kube-api-access-w2j7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:50.602003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.601983 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-provider-credentials" pod "62c78436-f731-4d09-9d46-cfa4540c6b08" (UID: "62c78436-f731-4d09-9d46-cfa4540c6b08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:00:50.702402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.702367 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2j7v\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-kube-api-access-w2j7v\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:50.702402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.702401 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/62c78436-f731-4d09-9d46-cfa4540c6b08-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:00:50.869160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.869126 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh"] Oct 31 22:00:50.872630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:50.872605 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545ffwxh"] Oct 31 22:00:51.224034 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.223958 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7"] Oct 31 22:00:51.224619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224598 4294 cpu_manager.go:414] "containerMap: removing container" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:51.224619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224620 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224643 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c977fc93-3845-44c2-a8b8-3bdafbcd2656" containerName="payload-processing" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224652 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c977fc93-3845-44c2-a8b8-3bdafbcd2656" containerName="payload-processing" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224660 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224668 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224678 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224683 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224691 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224695 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224716 4294 cpu_manager.go:414] "containerMap: removing container" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224723 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224734 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224740 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224749 4294 cpu_manager.go:414] "containerMap: removing container" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:51.224746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224755 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95452683-4ed9-4672-9026-339d1a0a07e5" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224767 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224772 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad066037-9e1a-48be-9db0-f3a9f8768a6b" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224778 4294 cpu_manager.go:414] "containerMap: removing container" podUID="62c78436-f731-4d09-9d46-cfa4540c6b08" containerName="payload-processing" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224782 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c78436-f731-4d09-9d46-cfa4540c6b08" containerName="payload-processing" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224792 4294 cpu_manager.go:414] "containerMap: removing container" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224796 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224803 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224807 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224814 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e808011d-7e69-4029-b11f-d22c647262e3" containerName="revoke-keys" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224818 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e808011d-7e69-4029-b11f-d22c647262e3" containerName="revoke-keys" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224827 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224831 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224927 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c977fc93-3845-44c2-a8b8-3bdafbcd2656" containerName="payload-processing" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224937 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3be5fe-e715-4121-bd7b-e76b9fb6edbd" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224943 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9b968b4-b2d0-4bc7-a8c1-d58735388295" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224950 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eae5971-48ba-41ff-bc32-4580174bd6c2" containerName="istio-proxy" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224956 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="decb8099-9e1b-4e62-8641-9769c2c9c83d" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224961 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0115189e-cf8e-4af0-b4d6-da4d82bd9db0" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224968 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ef7678-8bf6-443c-90e1-8c1af8de24d7" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224973 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2237055a-125a-45a4-998f-1ce799325f97" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224979 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="46a37b22-93fb-4781-a126-6a138e8ba5c7" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224986 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c78436-f731-4d09-9d46-cfa4540c6b08" containerName="payload-processing" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224993 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9ff7be-fa18-4a26-83fc-8e6f88ff1713" containerName="cleanup" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.224999 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e808011d-7e69-4029-b11f-d22c647262e3" containerName="revoke-keys" Oct 31 22:00:51.225171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.225006 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b742073-6765-4c13-931d-245b538180f6" containerName="cleanup" Oct 31 22:00:51.229676 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.229658 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.244582 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.244560 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"routing-overlay\"" Oct 31 22:00:51.244702 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.244637 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"kube-root-ca.crt\"" Oct 31 22:00:51.244702 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.244650 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"openshift-service-ca.crt\"" Oct 31 22:00:51.244702 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.244647 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"agc-cred-ed5d4be8dd27\"" Oct 31 22:00:51.244702 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.244673 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-696f9a\"" Oct 31 22:00:51.246563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.246542 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7"] Oct 31 22:00:51.254753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.254716 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-e2e--956cda1a-dockercfg-rtxsz\"" Oct 31 22:00:51.310020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.309993 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.310149 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.310048 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlln\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-kube-api-access-bdlln\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.310192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.310150 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.310291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.310275 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.416325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.416045 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.416325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.416152 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlln\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-kube-api-access-bdlln\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.416325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.416195 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.416634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.416372 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.422328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.418270 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.449322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.440371 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.449322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.445834 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.472047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.470895 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlln\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-kube-api-access-bdlln\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.544197 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.544166 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:51.825142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.825063 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn"] Oct 31 22:00:51.830187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.830168 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:51.845138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.845117 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"agc-cred-71ae21cabaf6\"" Oct 31 22:00:51.849613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.849589 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn"] Oct 31 22:00:51.861410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.861379 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7"] Oct 31 22:00:51.862094 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:51.862070 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0262fc84_f42d_417a_8119_f09738a6d722.slice/crio-27fdef9a50221e31f9af853bf67df0c05ea7ee34f8817c0cdcd00d4e3534b6c0 WatchSource:0}: Error finding container 27fdef9a50221e31f9af853bf67df0c05ea7ee34f8817c0cdcd00d4e3534b6c0: Status 404 returned error can't find the container with id 27fdef9a50221e31f9af853bf67df0c05ea7ee34f8817c0cdcd00d4e3534b6c0 Oct 31 22:00:51.925737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.925712 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnnzf\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-kube-api-access-hnnzf\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:51.925833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.925770 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:51.925927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.925911 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:51.926016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.926001 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:51.986297 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:51.986268 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c78436-f731-4d09-9d46-cfa4540c6b08" path="/var/lib/kubelet/pods/62c78436-f731-4d09-9d46-cfa4540c6b08/volumes" Oct 31 22:00:52.027060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.027036 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.027230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.027111 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.027230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.027222 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.027442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.027422 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnnzf\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-kube-api-access-hnnzf\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.027837 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.027814 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.027914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.027877 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.029555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.029538 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.050469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.050445 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnnzf\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-kube-api-access-hnnzf\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.145550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.145449 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.448149 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.448117 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn"] Oct 31 22:00:52.448611 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:00:52.448459 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df2ebf3_7b3f_49f9_97da_a889523e2b51.slice/crio-e6a36ece321fda71c36e3f5b9aebde549d74fedcad42226e039826ad50208a16 WatchSource:0}: Error finding container e6a36ece321fda71c36e3f5b9aebde549d74fedcad42226e039826ad50208a16: Status 404 returned error can't find the container with id e6a36ece321fda71c36e3f5b9aebde549d74fedcad42226e039826ad50208a16 Oct 31 22:00:52.535997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.535966 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" event={"ID":"0262fc84-f42d-417a-8119-f09738a6d722","Type":"ContainerStarted","Data":"317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8"} Oct 31 22:00:52.536116 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.536026 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" event={"ID":"0262fc84-f42d-417a-8119-f09738a6d722","Type":"ContainerStarted","Data":"27fdef9a50221e31f9af853bf67df0c05ea7ee34f8817c0cdcd00d4e3534b6c0"} Oct 31 22:00:52.536116 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.536056 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:00:52.537740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.537719 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" event={"ID":"0df2ebf3-7b3f-49f9-97da-a889523e2b51","Type":"ContainerStarted","Data":"f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce"} Oct 31 22:00:52.537838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.537764 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:00:52.537838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.537779 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" event={"ID":"0df2ebf3-7b3f-49f9-97da-a889523e2b51","Type":"ContainerStarted","Data":"e6a36ece321fda71c36e3f5b9aebde549d74fedcad42226e039826ad50208a16"} Oct 31 22:00:52.580480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.580433 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" podStartSLOduration=1.5804174720000002 podStartE2EDuration="1.580417472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:52.576430783 +0000 UTC m=+2692.701538012" watchObservedRunningTime="2026-10-31 22:00:52.580417472 +0000 UTC m=+2692.705524703" Oct 31 22:00:52.620972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:00:52.620913 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" podStartSLOduration=1.6208973850000001 podStartE2EDuration="1.620897385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:00:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:00:52.617567421 +0000 UTC m=+2692.742674652" watchObservedRunningTime="2026-10-31 22:00:52.620897385 +0000 UTC m=+2692.746004618" Oct 31 22:01:00.489324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:00.488292 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:01:00.518325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:00.512145 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:01:01.311594 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:01.311555 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:01:02.465382 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:02.465352 4294 scope.go:122] "RemoveContainer" containerID="a5907cbae4f1c7369eb9cdf299b4bc3a8d0b9de8d1aa8a8cf865a69de4edc2ed" Oct 31 22:01:02.475488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:02.475466 4294 scope.go:122] "RemoveContainer" containerID="fe2911d88d83e9faa7507d5a9cc19adaa8a766f4ff270edb3efd378c74d80eed" Oct 31 22:01:03.548434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:03.548400 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:01:03.548885 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:03.548562 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:01:03.624872 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:03.624838 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7"] Oct 31 22:01:08.625335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.625267 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" podUID="0262fc84-f42d-417a-8119-f09738a6d722" containerName="payload-processing" containerID="cri-o://317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8" gracePeriod=25 Oct 31 22:01:08.886681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.886625 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:01:08.963104 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.963065 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-plugins-config-volume\") pod \"0262fc84-f42d-417a-8119-f09738a6d722\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " Oct 31 22:01:08.963104 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.963110 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-provider-credentials\") pod \"0262fc84-f42d-417a-8119-f09738a6d722\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " Oct 31 22:01:08.963388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.963170 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-kube-api-access-bdlln\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-kube-api-access-bdlln\") pod \"0262fc84-f42d-417a-8119-f09738a6d722\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " Oct 31 22:01:08.963388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.963215 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-routing-overlay-volume\") pod \"0262fc84-f42d-417a-8119-f09738a6d722\" (UID: \"0262fc84-f42d-417a-8119-f09738a6d722\") " Oct 31 22:01:08.963619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.963589 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-plugins-config-volume" pod "0262fc84-f42d-417a-8119-f09738a6d722" (UID: "0262fc84-f42d-417a-8119-f09738a6d722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:08.963736 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.963704 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-routing-overlay-volume" pod "0262fc84-f42d-417a-8119-f09738a6d722" (UID: "0262fc84-f42d-417a-8119-f09738a6d722"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:08.964107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.964084 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:08.964184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.964113 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0262fc84-f42d-417a-8119-f09738a6d722-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:08.965665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.965637 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-provider-credentials" pod "0262fc84-f42d-417a-8119-f09738a6d722" (UID: "0262fc84-f42d-417a-8119-f09738a6d722"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:08.965797 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:08.965777 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-kube-api-access-bdlln" pod "0262fc84-f42d-417a-8119-f09738a6d722" (UID: "0262fc84-f42d-417a-8119-f09738a6d722"). InnerVolumeSpecName "kube-api-access-bdlln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:09.065085 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.065044 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:09.065085 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.065078 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdlln\" (UniqueName: \"kubernetes.io/projected/0262fc84-f42d-417a-8119-f09738a6d722-kube-api-access-bdlln\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:09.534510 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.534475 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv"] Oct 31 22:01:09.535111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.535092 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0262fc84-f42d-417a-8119-f09738a6d722" containerName="payload-processing" Oct 31 22:01:09.535164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.535113 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0262fc84-f42d-417a-8119-f09738a6d722" containerName="payload-processing" Oct 31 22:01:09.535204 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.535201 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0262fc84-f42d-417a-8119-f09738a6d722" containerName="payload-processing" Oct 31 22:01:09.538860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.538844 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.556023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.555990 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv"] Oct 31 22:01:09.564135 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.564106 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:01:09.564291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.564192 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:01:09.572043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.572018 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.572161 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.572077 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.572230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.572170 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4s8s\" (UniqueName: \"kubernetes.io/projected/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-kube-api-access-g4s8s\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.673126 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.673092 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.673568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.673178 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.673568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.673263 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4s8s\" (UniqueName: \"kubernetes.io/projected/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-kube-api-access-g4s8s\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.673860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.673839 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.675738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.675713 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.675865 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.675812 4294 generic.go:356] "Generic (PLEG): container finished" podID="0262fc84-f42d-417a-8119-f09738a6d722" containerID="317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8" exitCode=0 Oct 31 22:01:09.675865 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.675855 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" Oct 31 22:01:09.675943 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.675893 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" event={"ID":"0262fc84-f42d-417a-8119-f09738a6d722","Type":"ContainerDied","Data":"317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8"} Oct 31 22:01:09.675979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.675946 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7" event={"ID":"0262fc84-f42d-417a-8119-f09738a6d722","Type":"ContainerDied","Data":"27fdef9a50221e31f9af853bf67df0c05ea7ee34f8817c0cdcd00d4e3534b6c0"} Oct 31 22:01:09.675979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.675972 4294 scope.go:122] "RemoveContainer" containerID="317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8" Oct 31 22:01:09.692250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.692231 4294 scope.go:122] "RemoveContainer" containerID="317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8" Oct 31 22:01:09.692533 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:09.692510 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8\": container with ID starting with 317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8 not found: ID does not exist" containerID="317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8" Oct 31 22:01:09.692641 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.692537 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8"} err="failed to get container status \"317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8\": rpc error: code = NotFound desc = could not find container \"317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8\": container with ID starting with 317a204452b6f501b565aefa6e9ea219b45aabc0676903a2bdb97456911318a8 not found: ID does not exist" Oct 31 22:01:09.696362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.696290 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4s8s\" (UniqueName: \"kubernetes.io/projected/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-kube-api-access-g4s8s\") pod \"maas-api-revoke-sub-737cc848-crdsv\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.730920 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.730885 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7"] Oct 31 22:01:09.738704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.737704 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9nspj7"] Oct 31 22:01:09.853558 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.853469 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:09.990147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:09.990113 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0262fc84-f42d-417a-8119-f09738a6d722" path="/var/lib/kubelet/pods/0262fc84-f42d-417a-8119-f09738a6d722/volumes" Oct 31 22:01:10.185053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:10.185026 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv"] Oct 31 22:01:10.186081 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:10.186056 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519e661e_03ce_4b9c_87b2_21d26ce2d3fb.slice/crio-e437800f53903e197a5426f6acf5b617bec48297b210188e27c4ffa89f62f7e1 WatchSource:0}: Error finding container e437800f53903e197a5426f6acf5b617bec48297b210188e27c4ffa89f62f7e1: Status 404 returned error can't find the container with id e437800f53903e197a5426f6acf5b617bec48297b210188e27c4ffa89f62f7e1 Oct 31 22:01:10.685992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:10.685953 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" event={"ID":"519e661e-03ce-4b9c-87b2-21d26ce2d3fb","Type":"ContainerStarted","Data":"c86b0f868e0a196fd238f4cbfba4a6d8fa0f23600170b034ed5e80c4d88a5416"} Oct 31 22:01:10.686480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:10.686022 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" event={"ID":"519e661e-03ce-4b9c-87b2-21d26ce2d3fb","Type":"ContainerStarted","Data":"e437800f53903e197a5426f6acf5b617bec48297b210188e27c4ffa89f62f7e1"} Oct 31 22:01:10.726923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:10.726873 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" podStartSLOduration=1.726859897 podStartE2EDuration="1.726859897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:10.72582952 +0000 UTC m=+2710.850936751" watchObservedRunningTime="2026-10-31 22:01:10.726859897 +0000 UTC m=+2710.851967127" Oct 31 22:01:11.696075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:11.696041 4294 generic.go:356] "Generic (PLEG): container finished" podID="519e661e-03ce-4b9c-87b2-21d26ce2d3fb" containerID="c86b0f868e0a196fd238f4cbfba4a6d8fa0f23600170b034ed5e80c4d88a5416" exitCode=0 Oct 31 22:01:11.696527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:11.696121 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" event={"ID":"519e661e-03ce-4b9c-87b2-21d26ce2d3fb","Type":"ContainerDied","Data":"c86b0f868e0a196fd238f4cbfba4a6d8fa0f23600170b034ed5e80c4d88a5416"} Oct 31 22:01:12.861853 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.861828 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:12.926679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.926651 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-kube-api-access-g4s8s\" (UniqueName: \"kubernetes.io/projected/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-kube-api-access-g4s8s\") pod \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " Oct 31 22:01:12.926836 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.926772 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-xks-ca\") pod \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " Oct 31 22:01:12.926876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.926862 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-openshift-ca\") pod \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\" (UID: \"519e661e-03ce-4b9c-87b2-21d26ce2d3fb\") " Oct 31 22:01:12.927291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.927256 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-openshift-ca" pod "519e661e-03ce-4b9c-87b2-21d26ce2d3fb" (UID: "519e661e-03ce-4b9c-87b2-21d26ce2d3fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:12.927572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.927551 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:12.928707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.928673 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-kube-api-access-g4s8s" pod "519e661e-03ce-4b9c-87b2-21d26ce2d3fb" (UID: "519e661e-03ce-4b9c-87b2-21d26ce2d3fb"). InnerVolumeSpecName "kube-api-access-g4s8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:12.928800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:12.928733 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-xks-ca" pod "519e661e-03ce-4b9c-87b2-21d26ce2d3fb" (UID: "519e661e-03ce-4b9c-87b2-21d26ce2d3fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:01:13.028345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:13.028288 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4s8s\" (UniqueName: \"kubernetes.io/projected/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-kube-api-access-g4s8s\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:13.028473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:13.028346 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/519e661e-03ce-4b9c-87b2-21d26ce2d3fb-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:13.711446 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:13.711414 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" Oct 31 22:01:13.711604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:13.711451 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv" event={"ID":"519e661e-03ce-4b9c-87b2-21d26ce2d3fb","Type":"ContainerDied","Data":"e437800f53903e197a5426f6acf5b617bec48297b210188e27c4ffa89f62f7e1"} Oct 31 22:01:13.711604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:13.711495 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e437800f53903e197a5426f6acf5b617bec48297b210188e27c4ffa89f62f7e1" Oct 31 22:01:21.450023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.449982 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7"] Oct 31 22:01:21.450845 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.450823 4294 cpu_manager.go:414] "containerMap: removing container" podUID="519e661e-03ce-4b9c-87b2-21d26ce2d3fb" containerName="revoke-keys" Oct 31 22:01:21.450948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.450845 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519e661e-03ce-4b9c-87b2-21d26ce2d3fb" containerName="revoke-keys" Oct 31 22:01:21.451005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.450990 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="519e661e-03ce-4b9c-87b2-21d26ce2d3fb" containerName="revoke-keys" Oct 31 22:01:21.454996 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.454974 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.504336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.503618 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.504336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.503686 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.504336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.503845 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzbw2\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-kube-api-access-nzbw2\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.504336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.503901 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.507352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.505687 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"payload-processing-external-model-e2e--c7d64253-dockercfg-lhtkj\"" Oct 31 22:01:21.507352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.505845 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"routing-overlay\"" Oct 31 22:01:21.507352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.506497 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"kube-root-ca.crt\"" Oct 31 22:01:21.507352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.506567 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4bd51b\"" Oct 31 22:01:21.507352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.506629 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"e2e-cfl-a73b87-provider-cred\"" Oct 31 22:01:21.507352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.506703 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4bd51b\"/\"openshift-service-ca.crt\"" Oct 31 22:01:21.549333 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.542167 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7"] Oct 31 22:01:21.605067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.605033 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzbw2\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-kube-api-access-nzbw2\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.632922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.632888 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.635417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.618366 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv"] Oct 31 22:01:21.636979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.636953 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.638800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.635394 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.639400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.639373 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.640483 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.640177 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.645400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.642762 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.647276 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.647112 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzbw2\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-kube-api-access-nzbw2\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.648965 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.648932 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.669371 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.669345 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:01:21.678728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.678700 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv"] Oct 31 22:01:21.741781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.741747 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.741991 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.741817 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.741991 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.741931 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zsr\" (UniqueName: \"kubernetes.io/projected/c744edf9-814d-46d1-aeda-ae19ac026ebc-kube-api-access-m4zsr\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.760599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.760573 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:01:21.828255 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.828209 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:21.843702 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.843657 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.843842 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.843741 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.843842 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.843796 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zsr\" (UniqueName: \"kubernetes.io/projected/c744edf9-814d-46d1-aeda-ae19ac026ebc-kube-api-access-m4zsr\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.844436 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.844414 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.846825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.846802 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.867178 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.867141 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zsr\" (UniqueName: \"kubernetes.io/projected/c744edf9-814d-46d1-aeda-ae19ac026ebc-kube-api-access-m4zsr\") pod \"maas-api-revoke-sub-da1c079c-vkjzv\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:21.992877 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:21.992792 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:22.142629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.142594 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7"] Oct 31 22:01:22.143030 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:22.143000 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecb9572_b09e_43a8_9117_87c6b2976552.slice/crio-7fdba6294102551123027c9b46236cc42bd856b6ade787b2ddf6e18c9a7a6511 WatchSource:0}: Error finding container 7fdba6294102551123027c9b46236cc42bd856b6ade787b2ddf6e18c9a7a6511: Status 404 returned error can't find the container with id 7fdba6294102551123027c9b46236cc42bd856b6ade787b2ddf6e18c9a7a6511 Oct 31 22:01:22.354488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.354457 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv"] Oct 31 22:01:22.356288 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:22.356264 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc744edf9_814d_46d1_aeda_ae19ac026ebc.slice/crio-de68bec8f110233c090f5dd7cc70816221296c70c95d49547736b732e256814c WatchSource:0}: Error finding container de68bec8f110233c090f5dd7cc70816221296c70c95d49547736b732e256814c: Status 404 returned error can't find the container with id de68bec8f110233c090f5dd7cc70816221296c70c95d49547736b732e256814c Oct 31 22:01:22.774079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.774042 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" event={"ID":"c744edf9-814d-46d1-aeda-ae19ac026ebc","Type":"ContainerStarted","Data":"b61a635795a66fd5eb692e3c5e23ae9e4422bcb383a3baefa108363be58dec76"} Oct 31 22:01:22.774613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.774106 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" event={"ID":"c744edf9-814d-46d1-aeda-ae19ac026ebc","Type":"ContainerStarted","Data":"de68bec8f110233c090f5dd7cc70816221296c70c95d49547736b732e256814c"} Oct 31 22:01:22.775703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.775676 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" event={"ID":"2ecb9572-b09e-43a8-9117-87c6b2976552","Type":"ContainerStarted","Data":"038e0576ef18b8edaa0c304866a68d26fb01691df6e0cd4901b2487cd0150cb6"} Oct 31 22:01:22.775799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.775726 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" event={"ID":"2ecb9572-b09e-43a8-9117-87c6b2976552","Type":"ContainerStarted","Data":"7fdba6294102551123027c9b46236cc42bd856b6ade787b2ddf6e18c9a7a6511"} Oct 31 22:01:22.775864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.775802 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:22.817617 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.817568 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" podStartSLOduration=1.8175551300000001 podStartE2EDuration="1.81755513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:22.81450024 +0000 UTC m=+2722.939607470" watchObservedRunningTime="2026-10-31 22:01:22.81755513 +0000 UTC m=+2722.942662360" Oct 31 22:01:22.856562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:22.856474 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" podStartSLOduration=1.85646223 podStartE2EDuration="1.85646223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:22.855557256 +0000 UTC m=+2722.980664500" watchObservedRunningTime="2026-10-31 22:01:22.85646223 +0000 UTC m=+2722.981569472" Oct 31 22:01:23.792044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.791992 4294 generic.go:356] "Generic (PLEG): container finished" podID="c744edf9-814d-46d1-aeda-ae19ac026ebc" containerID="b61a635795a66fd5eb692e3c5e23ae9e4422bcb383a3baefa108363be58dec76" exitCode=0 Oct 31 22:01:23.793621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.793352 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" event={"ID":"c744edf9-814d-46d1-aeda-ae19ac026ebc","Type":"ContainerDied","Data":"b61a635795a66fd5eb692e3c5e23ae9e4422bcb383a3baefa108363be58dec76"} Oct 31 22:01:23.927327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.927032 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp"] Oct 31 22:01:23.931565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.931542 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:23.932879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.932852 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7"] Oct 31 22:01:23.938000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.937964 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:23.952976 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.952899 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp"] Oct 31 22:01:23.957093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:23.957067 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7"] Oct 31 22:01:24.087484 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.087376 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.087484 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.087424 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m2wf\" (UniqueName: \"kubernetes.io/projected/c3f125d6-ccbf-4cb0-8e6d-e37329991737-kube-api-access-2m2wf\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.087737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.087622 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.087795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.087739 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.087876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.087857 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/e83c35d0-8de3-4def-a6ea-032dbd5362d7-kube-api-access-pbggz\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.087984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.087965 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.189022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.188986 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/e83c35d0-8de3-4def-a6ea-032dbd5362d7-kube-api-access-pbggz\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.189241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.189041 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.189241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.189107 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.189241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.189123 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m2wf\" (UniqueName: \"kubernetes.io/projected/c3f125d6-ccbf-4cb0-8e6d-e37329991737-kube-api-access-2m2wf\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.189241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.189202 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.189509 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.189327 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.189973 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.189940 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.190097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.189984 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.191565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.191536 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.191743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.191724 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.213138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.213106 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m2wf\" (UniqueName: \"kubernetes.io/projected/c3f125d6-ccbf-4cb0-8e6d-e37329991737-kube-api-access-2m2wf\") pod \"maas-api-revoke-sub-f11a5a9f-svwcp\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.213409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.213371 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/e83c35d0-8de3-4def-a6ea-032dbd5362d7-kube-api-access-pbggz\") pod \"maas-api-revoke-sub-1f1d87a8-8hvf7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.247180 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.247152 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:24.264792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.264764 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:24.552640 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.552608 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp"] Oct 31 22:01:24.553553 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:24.553525 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f125d6_ccbf_4cb0_8e6d_e37329991737.slice/crio-f8931c912be2c231a90271842ebf776651039c6f3520876f1dee7d16bbba5753 WatchSource:0}: Error finding container f8931c912be2c231a90271842ebf776651039c6f3520876f1dee7d16bbba5753: Status 404 returned error can't find the container with id f8931c912be2c231a90271842ebf776651039c6f3520876f1dee7d16bbba5753 Oct 31 22:01:24.584561 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.584534 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7"] Oct 31 22:01:24.589654 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:24.589626 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83c35d0_8de3_4def_a6ea_032dbd5362d7.slice/crio-fdbb4acaaf6d6f360543836549b5508670a527724942a25c64acd26419ebf195 WatchSource:0}: Error finding container fdbb4acaaf6d6f360543836549b5508670a527724942a25c64acd26419ebf195: Status 404 returned error can't find the container with id fdbb4acaaf6d6f360543836549b5508670a527724942a25c64acd26419ebf195 Oct 31 22:01:24.801715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.801680 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" event={"ID":"c3f125d6-ccbf-4cb0-8e6d-e37329991737","Type":"ContainerStarted","Data":"f8931c912be2c231a90271842ebf776651039c6f3520876f1dee7d16bbba5753"} Oct 31 22:01:24.803220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:24.803186 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" event={"ID":"e83c35d0-8de3-4def-a6ea-032dbd5362d7","Type":"ContainerStarted","Data":"fdbb4acaaf6d6f360543836549b5508670a527724942a25c64acd26419ebf195"} Oct 31 22:01:25.081443 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.081419 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:25.201844 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.201756 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c744edf9-814d-46d1-aeda-ae19ac026ebc-kube-api-access-m4zsr\" (UniqueName: \"kubernetes.io/projected/c744edf9-814d-46d1-aeda-ae19ac026ebc-kube-api-access-m4zsr\") pod \"c744edf9-814d-46d1-aeda-ae19ac026ebc\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " Oct 31 22:01:25.201844 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.201803 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-openshift-ca\") pod \"c744edf9-814d-46d1-aeda-ae19ac026ebc\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " Oct 31 22:01:25.202087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.202009 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-xks-ca\") pod \"c744edf9-814d-46d1-aeda-ae19ac026ebc\" (UID: \"c744edf9-814d-46d1-aeda-ae19ac026ebc\") " Oct 31 22:01:25.202217 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.202193 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-openshift-ca" pod "c744edf9-814d-46d1-aeda-ae19ac026ebc" (UID: "c744edf9-814d-46d1-aeda-ae19ac026ebc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:25.202572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.202552 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:25.203985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.203954 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c744edf9-814d-46d1-aeda-ae19ac026ebc-kube-api-access-m4zsr" pod "c744edf9-814d-46d1-aeda-ae19ac026ebc" (UID: "c744edf9-814d-46d1-aeda-ae19ac026ebc"). InnerVolumeSpecName "kube-api-access-m4zsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:25.204128 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.203959 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-xks-ca" pod "c744edf9-814d-46d1-aeda-ae19ac026ebc" (UID: "c744edf9-814d-46d1-aeda-ae19ac026ebc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:01:25.303648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.303611 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zsr\" (UniqueName: \"kubernetes.io/projected/c744edf9-814d-46d1-aeda-ae19ac026ebc-kube-api-access-m4zsr\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:25.303648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.303647 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c744edf9-814d-46d1-aeda-ae19ac026ebc-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:25.813128 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.813041 4294 generic.go:356] "Generic (PLEG): container finished" podID="c3f125d6-ccbf-4cb0-8e6d-e37329991737" containerID="b19358c898d51a52e09d826b75c539a6aa2be5301b6ca033e7ac55589f2344da" exitCode=0 Oct 31 22:01:25.813627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.813124 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" event={"ID":"c3f125d6-ccbf-4cb0-8e6d-e37329991737","Type":"ContainerDied","Data":"b19358c898d51a52e09d826b75c539a6aa2be5301b6ca033e7ac55589f2344da"} Oct 31 22:01:25.814762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.814738 4294 generic.go:356] "Generic (PLEG): container finished" podID="e83c35d0-8de3-4def-a6ea-032dbd5362d7" containerID="a3b577c5c40d8aee838cf803131ec08f06e2aba093d3f8a058f482448bcb012e" exitCode=0 Oct 31 22:01:25.814861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.814809 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" event={"ID":"e83c35d0-8de3-4def-a6ea-032dbd5362d7","Type":"ContainerDied","Data":"a3b577c5c40d8aee838cf803131ec08f06e2aba093d3f8a058f482448bcb012e"} Oct 31 22:01:25.816002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.815986 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" Oct 31 22:01:25.816068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.816007 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv" event={"ID":"c744edf9-814d-46d1-aeda-ae19ac026ebc","Type":"ContainerDied","Data":"de68bec8f110233c090f5dd7cc70816221296c70c95d49547736b732e256814c"} Oct 31 22:01:25.816068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:25.816036 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de68bec8f110233c090f5dd7cc70816221296c70c95d49547736b732e256814c" Oct 31 22:01:26.987398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:26.987363 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:27.027890 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.027864 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:27.131821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.131732 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3f125d6-ccbf-4cb0-8e6d-e37329991737-kube-api-access-2m2wf\" (UniqueName: \"kubernetes.io/projected/c3f125d6-ccbf-4cb0-8e6d-e37329991737-kube-api-access-2m2wf\") pod \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " Oct 31 22:01:27.131821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.131818 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83c35d0-8de3-4def-a6ea-032dbd5362d7-kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/e83c35d0-8de3-4def-a6ea-032dbd5362d7-kube-api-access-pbggz\") pod \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " Oct 31 22:01:27.132057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.131842 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-openshift-ca\") pod \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " Oct 31 22:01:27.132057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.131999 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-openshift-ca\") pod \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " Oct 31 22:01:27.132057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.132028 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-xks-ca\") pod \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\" (UID: \"e83c35d0-8de3-4def-a6ea-032dbd5362d7\") " Oct 31 22:01:27.132226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.132061 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-xks-ca\") pod \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\" (UID: \"c3f125d6-ccbf-4cb0-8e6d-e37329991737\") " Oct 31 22:01:27.132367 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.132332 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-openshift-ca" pod "c3f125d6-ccbf-4cb0-8e6d-e37329991737" (UID: "c3f125d6-ccbf-4cb0-8e6d-e37329991737"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:27.132517 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.132481 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-openshift-ca" pod "e83c35d0-8de3-4def-a6ea-032dbd5362d7" (UID: "e83c35d0-8de3-4def-a6ea-032dbd5362d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:27.132986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.132961 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:27.133106 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.132990 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:27.134207 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.134180 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-xks-ca" pod "e83c35d0-8de3-4def-a6ea-032dbd5362d7" (UID: "e83c35d0-8de3-4def-a6ea-032dbd5362d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:01:27.134287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.134206 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3f125d6-ccbf-4cb0-8e6d-e37329991737-kube-api-access-2m2wf" pod "c3f125d6-ccbf-4cb0-8e6d-e37329991737" (UID: "c3f125d6-ccbf-4cb0-8e6d-e37329991737"). InnerVolumeSpecName "kube-api-access-2m2wf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:27.134287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.134230 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83c35d0-8de3-4def-a6ea-032dbd5362d7-kube-api-access-pbggz" pod "e83c35d0-8de3-4def-a6ea-032dbd5362d7" (UID: "e83c35d0-8de3-4def-a6ea-032dbd5362d7"). InnerVolumeSpecName "kube-api-access-pbggz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:27.134457 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.134295 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-xks-ca" pod "c3f125d6-ccbf-4cb0-8e6d-e37329991737" (UID: "c3f125d6-ccbf-4cb0-8e6d-e37329991737"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:01:27.234143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.234109 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83c35d0-8de3-4def-a6ea-032dbd5362d7-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:27.234143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.234140 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3f125d6-ccbf-4cb0-8e6d-e37329991737-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:27.234143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.234150 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m2wf\" (UniqueName: \"kubernetes.io/projected/c3f125d6-ccbf-4cb0-8e6d-e37329991737-kube-api-access-2m2wf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:27.234429 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.234161 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbggz\" (UniqueName: \"kubernetes.io/projected/e83c35d0-8de3-4def-a6ea-032dbd5362d7-kube-api-access-pbggz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:27.830719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.830687 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" Oct 31 22:01:27.830954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.830722 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp" event={"ID":"c3f125d6-ccbf-4cb0-8e6d-e37329991737","Type":"ContainerDied","Data":"f8931c912be2c231a90271842ebf776651039c6f3520876f1dee7d16bbba5753"} Oct 31 22:01:27.830954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.830783 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8931c912be2c231a90271842ebf776651039c6f3520876f1dee7d16bbba5753" Oct 31 22:01:27.833195 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.833179 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" Oct 31 22:01:27.833294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.833203 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7" event={"ID":"e83c35d0-8de3-4def-a6ea-032dbd5362d7","Type":"ContainerDied","Data":"fdbb4acaaf6d6f360543836549b5508670a527724942a25c64acd26419ebf195"} Oct 31 22:01:27.833294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:27.833240 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdbb4acaaf6d6f360543836549b5508670a527724942a25c64acd26419ebf195" Oct 31 22:01:31.310396 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:31.310357 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:01:32.766507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:32.766464 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7"] Oct 31 22:01:32.769994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:32.769968 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:33.440252 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:33.440203 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t"] Oct 31 22:01:33.440645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:33.440611 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" containerID="cri-o://a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e" gracePeriod=30 Oct 31 22:01:35.427608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.427573 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9"] Oct 31 22:01:35.428228 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428214 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e83c35d0-8de3-4def-a6ea-032dbd5362d7" containerName="revoke-keys" Oct 31 22:01:35.428274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428230 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83c35d0-8de3-4def-a6ea-032dbd5362d7" containerName="revoke-keys" Oct 31 22:01:35.428274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428253 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c3f125d6-ccbf-4cb0-8e6d-e37329991737" containerName="revoke-keys" Oct 31 22:01:35.428274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428257 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3f125d6-ccbf-4cb0-8e6d-e37329991737" containerName="revoke-keys" Oct 31 22:01:35.428409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428274 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c744edf9-814d-46d1-aeda-ae19ac026ebc" containerName="revoke-keys" Oct 31 22:01:35.428409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428280 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c744edf9-814d-46d1-aeda-ae19ac026ebc" containerName="revoke-keys" Oct 31 22:01:35.428409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428391 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83c35d0-8de3-4def-a6ea-032dbd5362d7" containerName="revoke-keys" Oct 31 22:01:35.428409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428402 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3f125d6-ccbf-4cb0-8e6d-e37329991737" containerName="revoke-keys" Oct 31 22:01:35.428409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.428411 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c744edf9-814d-46d1-aeda-ae19ac026ebc" containerName="revoke-keys" Oct 31 22:01:35.432171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.432151 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.450414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.450379 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9"] Oct 31 22:01:35.456437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.456378 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:01:35.456578 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.456539 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:01:35.493954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.493917 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044f0321-8883-432a-a4be-273877d04cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.494159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.493991 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xclgz\" (UniqueName: \"kubernetes.io/projected/044f0321-8883-432a-a4be-273877d04cc7-kube-api-access-xclgz\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.494234 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.494181 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044f0321-8883-432a-a4be-273877d04cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.595936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.595894 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xclgz\" (UniqueName: \"kubernetes.io/projected/044f0321-8883-432a-a4be-273877d04cc7-kube-api-access-xclgz\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.596102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.596055 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044f0321-8883-432a-a4be-273877d04cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.596277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.596258 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044f0321-8883-432a-a4be-273877d04cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.596778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.596754 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044f0321-8883-432a-a4be-273877d04cc7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.598682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.598663 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044f0321-8883-432a-a4be-273877d04cc7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.660254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.660211 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xclgz\" (UniqueName: \"kubernetes.io/projected/044f0321-8883-432a-a4be-273877d04cc7-kube-api-access-xclgz\") pod \"maas-api-revoke-sub-85171cb4-m5zs9\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:35.749222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:35.748892 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:36.105573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:36.105495 4294 patch_prober.go:29] interesting pod/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:01:36.105573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:36.105549 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:01:36.113185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:36.113162 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9"] Oct 31 22:01:36.113608 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:36.113587 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044f0321_8883_432a_a4be_273877d04cc7.slice/crio-685ec7bb3bfba653a02800fe167cceecca80e589b5a98a699227899e473a0ccf WatchSource:0}: Error finding container 685ec7bb3bfba653a02800fe167cceecca80e589b5a98a699227899e473a0ccf: Status 404 returned error can't find the container with id 685ec7bb3bfba653a02800fe167cceecca80e589b5a98a699227899e473a0ccf Oct 31 22:01:36.910678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:36.910637 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" event={"ID":"044f0321-8883-432a-a4be-273877d04cc7","Type":"ContainerStarted","Data":"e8a0017aa956a3b42dddeadef349785de71e5f9d58c27c29e878aedf3add8cf9"} Oct 31 22:01:36.911583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:36.910703 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" event={"ID":"044f0321-8883-432a-a4be-273877d04cc7","Type":"ContainerStarted","Data":"685ec7bb3bfba653a02800fe167cceecca80e589b5a98a699227899e473a0ccf"} Oct 31 22:01:36.955277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:36.955229 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" podStartSLOduration=1.9552133870000001 podStartE2EDuration="1.955213387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:36.953444934 +0000 UTC m=+2737.078552164" watchObservedRunningTime="2026-10-31 22:01:36.955213387 +0000 UTC m=+2737.080320632" Oct 31 22:01:37.767244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:37.767196 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" podUID="2ecb9572-b09e-43a8-9117-87c6b2976552" containerName="payload-processing" containerID="cri-o://038e0576ef18b8edaa0c304866a68d26fb01691df6e0cd4901b2487cd0150cb6" gracePeriod=25 Oct 31 22:01:37.929560 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:37.929524 4294 generic.go:356] "Generic (PLEG): container finished" podID="2ecb9572-b09e-43a8-9117-87c6b2976552" containerID="038e0576ef18b8edaa0c304866a68d26fb01691df6e0cd4901b2487cd0150cb6" exitCode=0 Oct 31 22:01:37.930002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:37.929605 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" event={"ID":"2ecb9572-b09e-43a8-9117-87c6b2976552","Type":"ContainerDied","Data":"038e0576ef18b8edaa0c304866a68d26fb01691df6e0cd4901b2487cd0150cb6"} Oct 31 22:01:37.931493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:37.931401 4294 generic.go:356] "Generic (PLEG): container finished" podID="044f0321-8883-432a-a4be-273877d04cc7" containerID="e8a0017aa956a3b42dddeadef349785de71e5f9d58c27c29e878aedf3add8cf9" exitCode=0 Oct 31 22:01:37.931493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:37.931432 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" event={"ID":"044f0321-8883-432a-a4be-273877d04cc7","Type":"ContainerDied","Data":"e8a0017aa956a3b42dddeadef349785de71e5f9d58c27c29e878aedf3add8cf9"} Oct 31 22:01:38.204860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.204828 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:38.257698 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.257668 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-routing-overlay-volume\") pod \"2ecb9572-b09e-43a8-9117-87c6b2976552\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " Oct 31 22:01:38.257899 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.257751 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-plugins-config-volume\") pod \"2ecb9572-b09e-43a8-9117-87c6b2976552\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " Oct 31 22:01:38.257966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.257949 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-provider-credentials\") pod \"2ecb9572-b09e-43a8-9117-87c6b2976552\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " Oct 31 22:01:38.258030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.257996 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-kube-api-access-nzbw2\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-kube-api-access-nzbw2\") pod \"2ecb9572-b09e-43a8-9117-87c6b2976552\" (UID: \"2ecb9572-b09e-43a8-9117-87c6b2976552\") " Oct 31 22:01:38.258143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.258113 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-routing-overlay-volume" pod "2ecb9572-b09e-43a8-9117-87c6b2976552" (UID: "2ecb9572-b09e-43a8-9117-87c6b2976552"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:38.258143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.258132 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-plugins-config-volume" pod "2ecb9572-b09e-43a8-9117-87c6b2976552" (UID: "2ecb9572-b09e-43a8-9117-87c6b2976552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:38.258591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.258570 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.258697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.258594 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2ecb9572-b09e-43a8-9117-87c6b2976552-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.260557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.260535 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-kube-api-access-nzbw2" pod "2ecb9572-b09e-43a8-9117-87c6b2976552" (UID: "2ecb9572-b09e-43a8-9117-87c6b2976552"). InnerVolumeSpecName "kube-api-access-nzbw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:38.260954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.260921 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-provider-credentials" pod "2ecb9572-b09e-43a8-9117-87c6b2976552" (UID: "2ecb9572-b09e-43a8-9117-87c6b2976552"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:38.359903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.359866 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.359903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.359896 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzbw2\" (UniqueName: \"kubernetes.io/projected/2ecb9572-b09e-43a8-9117-87c6b2976552-kube-api-access-nzbw2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.681798 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.681771 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.763921 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-podinfo\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764032 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-envoy\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764105 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-token\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-token\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764136 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-certs\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764209 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-data\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764249 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istiod-ca-cert\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764693 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-certs" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764852 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-credential-socket\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.764974 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-kube-api-access-2k65x\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-kube-api-access-2k65x\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765017 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-data" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:01:38.765215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765043 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-socket\") pod \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\" (UID: \"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9\") " Oct 31 22:01:38.766205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765241 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istiod-ca-cert" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:38.766205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765446 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-socket" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:01:38.766205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765718 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.766205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765768 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.766205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765783 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.766205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.765799 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.766567 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.766525 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-podinfo" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:01:38.766727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.766686 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-credential-socket" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:01:38.766866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.766828 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-token" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:38.767854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.767825 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-envoy" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:01:38.767934 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.767860 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-kube-api-access-2k65x" pod "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" (UID: "c89cdf40-1bb8-4ca0-b66b-2fc6178737f9"). InnerVolumeSpecName "kube-api-access-2k65x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:38.867419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.867384 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k65x\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-kube-api-access-2k65x\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.867419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.867414 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.867419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.867423 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.867419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.867431 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.867693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.867439 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:38.940222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.940186 4294 generic.go:356] "Generic (PLEG): container finished" podID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerID="a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e" exitCode=0 Oct 31 22:01:38.940721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.940244 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" Oct 31 22:01:38.940721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.940272 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" event={"ID":"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9","Type":"ContainerDied","Data":"a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e"} Oct 31 22:01:38.940721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.940355 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t" event={"ID":"c89cdf40-1bb8-4ca0-b66b-2fc6178737f9","Type":"ContainerDied","Data":"4236379054d06578e541aae734f2ef870c8f651f5443f88326d87277038beae6"} Oct 31 22:01:38.940721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.940383 4294 scope.go:122] "RemoveContainer" containerID="a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e" Oct 31 22:01:38.942079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.942055 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" Oct 31 22:01:38.942162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.942094 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7" event={"ID":"2ecb9572-b09e-43a8-9117-87c6b2976552","Type":"ContainerDied","Data":"7fdba6294102551123027c9b46236cc42bd856b6ade787b2ddf6e18c9a7a6511"} Oct 31 22:01:38.953165 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.953140 4294 scope.go:122] "RemoveContainer" containerID="a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e" Oct 31 22:01:38.953516 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:38.953483 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e\": container with ID starting with a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e not found: ID does not exist" containerID="a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e" Oct 31 22:01:38.953603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.953522 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e"} err="failed to get container status \"a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e\": rpc error: code = NotFound desc = could not find container \"a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e\": container with ID starting with a243d14388d418f6a99a15c2d679257cd389ceaa2f78e4564fa51ea60fad6a3e not found: ID does not exist" Oct 31 22:01:38.953603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:38.953538 4294 scope.go:122] "RemoveContainer" containerID="038e0576ef18b8edaa0c304866a68d26fb01691df6e0cd4901b2487cd0150cb6" Oct 31 22:01:39.074394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.071379 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t"] Oct 31 22:01:39.089826 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.089790 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-106555-openshift-default-5bdb5c6bc9-fsm7t"] Oct 31 22:01:39.095938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.095910 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7"] Oct 31 22:01:39.100284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.100254 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4bd51b/payload-processing-external-model-e2e-extmod-a-4bd51b-5545hhfn7"] Oct 31 22:01:39.153878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.153837 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf"] Oct 31 22:01:39.154540 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.154522 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" Oct 31 22:01:39.154540 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.154540 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" Oct 31 22:01:39.154679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.154556 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2ecb9572-b09e-43a8-9117-87c6b2976552" containerName="payload-processing" Oct 31 22:01:39.154679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.154561 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecb9572-b09e-43a8-9117-87c6b2976552" containerName="payload-processing" Oct 31 22:01:39.154679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.154671 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" containerName="istio-proxy" Oct 31 22:01:39.154832 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.154685 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecb9572-b09e-43a8-9117-87c6b2976552" containerName="payload-processing" Oct 31 22:01:39.160109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.160079 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.181629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.181231 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"payload-processing-external-model-plugins-e2e-extmod-a-943963\"" Oct 31 22:01:39.181629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.181293 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"agc-cred-4513df987357\"" Oct 31 22:01:39.181629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.181486 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"routing-overlay\"" Oct 31 22:01:39.181629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.181525 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"openshift-service-ca.crt\"" Oct 31 22:01:39.181629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.181580 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"kube-root-ca.crt\"" Oct 31 22:01:39.182116 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.182095 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"payload-processing-external-model-e2e--1e97e1d3-dockercfg-bw68l\"" Oct 31 22:01:39.183471 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.183445 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb2db\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-kube-api-access-xb2db\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.183600 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.183537 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.183676 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.183598 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.183767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.183749 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.184501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.184477 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf"] Oct 31 22:01:39.217446 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.217417 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:39.284249 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.284218 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/044f0321-8883-432a-a4be-273877d04cc7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044f0321-8883-432a-a4be-273877d04cc7-maas-api-xks-ca\") pod \"044f0321-8883-432a-a4be-273877d04cc7\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " Oct 31 22:01:39.284445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.284391 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/044f0321-8883-432a-a4be-273877d04cc7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044f0321-8883-432a-a4be-273877d04cc7-maas-api-openshift-ca\") pod \"044f0321-8883-432a-a4be-273877d04cc7\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " Oct 31 22:01:39.284445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.284423 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/044f0321-8883-432a-a4be-273877d04cc7-kube-api-access-xclgz\" (UniqueName: \"kubernetes.io/projected/044f0321-8883-432a-a4be-273877d04cc7-kube-api-access-xclgz\") pod \"044f0321-8883-432a-a4be-273877d04cc7\" (UID: \"044f0321-8883-432a-a4be-273877d04cc7\") " Oct 31 22:01:39.284997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.284778 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/044f0321-8883-432a-a4be-273877d04cc7-maas-api-openshift-ca" pod "044f0321-8883-432a-a4be-273877d04cc7" (UID: "044f0321-8883-432a-a4be-273877d04cc7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:39.284997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.284786 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb2db\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-kube-api-access-xb2db\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.284997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.284914 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.284997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.284984 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.285297 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.285154 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.286185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.285427 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/044f0321-8883-432a-a4be-273877d04cc7-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:39.286185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.285988 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.286185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.286120 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.287047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.287019 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/044f0321-8883-432a-a4be-273877d04cc7-maas-api-xks-ca" pod "044f0321-8883-432a-a4be-273877d04cc7" (UID: "044f0321-8883-432a-a4be-273877d04cc7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:01:39.287699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.287675 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.288009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.287987 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/044f0321-8883-432a-a4be-273877d04cc7-kube-api-access-xclgz" pod "044f0321-8883-432a-a4be-273877d04cc7" (UID: "044f0321-8883-432a-a4be-273877d04cc7"). InnerVolumeSpecName "kube-api-access-xclgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:39.308194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.308172 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb2db\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-kube-api-access-xb2db\") pod \"payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.393909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.393449 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xclgz\" (UniqueName: \"kubernetes.io/projected/044f0321-8883-432a-a4be-273877d04cc7-kube-api-access-xclgz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:39.393909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.393493 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/044f0321-8883-432a-a4be-273877d04cc7-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:39.474983 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.474907 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.781004 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.780976 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf"] Oct 31 22:01:39.782379 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:39.782344 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00138ef9_1ba8_4578_b64a_72d00ee8bcdd.slice/crio-64fd127e6e9a7b6bed9de1a934973c32caf07536bc930a3858aa8aa32f9b54ca WatchSource:0}: Error finding container 64fd127e6e9a7b6bed9de1a934973c32caf07536bc930a3858aa8aa32f9b54ca: Status 404 returned error can't find the container with id 64fd127e6e9a7b6bed9de1a934973c32caf07536bc930a3858aa8aa32f9b54ca Oct 31 22:01:39.950100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.950064 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" event={"ID":"00138ef9-1ba8-4578-b64a-72d00ee8bcdd","Type":"ContainerStarted","Data":"b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8"} Oct 31 22:01:39.950573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.950119 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" event={"ID":"00138ef9-1ba8-4578-b64a-72d00ee8bcdd","Type":"ContainerStarted","Data":"64fd127e6e9a7b6bed9de1a934973c32caf07536bc930a3858aa8aa32f9b54ca"} Oct 31 22:01:39.950573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.950146 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:39.952473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.952451 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" Oct 31 22:01:39.952591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.952486 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9" event={"ID":"044f0321-8883-432a-a4be-273877d04cc7","Type":"ContainerDied","Data":"685ec7bb3bfba653a02800fe167cceecca80e589b5a98a699227899e473a0ccf"} Oct 31 22:01:39.952591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.952526 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="685ec7bb3bfba653a02800fe167cceecca80e589b5a98a699227899e473a0ccf" Oct 31 22:01:39.986479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.986455 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecb9572-b09e-43a8-9117-87c6b2976552" path="/var/lib/kubelet/pods/2ecb9572-b09e-43a8-9117-87c6b2976552/volumes" Oct 31 22:01:39.986976 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.986957 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89cdf40-1bb8-4ca0-b66b-2fc6178737f9" path="/var/lib/kubelet/pods/c89cdf40-1bb8-4ca0-b66b-2fc6178737f9/volumes" Oct 31 22:01:39.992795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:39.992754 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" podStartSLOduration=0.992738775 podStartE2EDuration="992.738775ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:39.990557152 +0000 UTC m=+2740.115664382" watchObservedRunningTime="2026-10-31 22:01:39.992738775 +0000 UTC m=+2740.117846008" Oct 31 22:01:47.197213 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.197136 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz"] Oct 31 22:01:47.197846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.197830 4294 cpu_manager.go:414] "containerMap: removing container" podUID="044f0321-8883-432a-a4be-273877d04cc7" containerName="revoke-keys" Oct 31 22:01:47.197912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.197847 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="044f0321-8883-432a-a4be-273877d04cc7" containerName="revoke-keys" Oct 31 22:01:47.197988 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.197977 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="044f0321-8883-432a-a4be-273877d04cc7" containerName="revoke-keys" Oct 31 22:01:47.203133 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.203112 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.220341 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.218854 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fac043\"/\"routing-overlay\"" Oct 31 22:01:47.220341 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.220143 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz"] Oct 31 22:01:47.225919 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.225894 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fac043\"/\"agc-cred-7cc44fcedad2\"" Oct 31 22:01:47.226152 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.226079 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fac043\"/\"agc-cred-b06b04d8d7ad\"" Oct 31 22:01:47.227569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.227541 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-fac043\"/\"payload-processing-external-model-e2e--2792e8e2-dockercfg-49svf\"" Oct 31 22:01:47.227654 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.227617 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fac043\"/\"kube-root-ca.crt\"" Oct 31 22:01:47.231136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.231112 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fac043\"/\"payload-processing-external-model-plugins-e2e-extmod-a-fac043\"" Oct 31 22:01:47.231285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.231256 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-fac043\"/\"openshift-service-ca.crt\"" Oct 31 22:01:47.309431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.309396 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.309614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.309463 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvq28\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-kube-api-access-tvq28\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.309698 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.309608 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.309777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.309757 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.410633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.410594 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.410820 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.410716 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.410872 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.410852 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.410931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.410920 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvq28\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-kube-api-access-tvq28\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.411545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.411513 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.411743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.411711 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.413475 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.413453 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.452373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.446362 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvq28\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-kube-api-access-tvq28\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.530281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.523551 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:47.878585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:47.878555 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz"] Oct 31 22:01:47.879186 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:47.879158 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c488cdd_c7ae_4f68_9ab7_717fd2cf8f69.slice/crio-1969e6eec251f4d51169ccca9aba8b100ae835a3dcc2b9fac30cc7557d037439 WatchSource:0}: Error finding container 1969e6eec251f4d51169ccca9aba8b100ae835a3dcc2b9fac30cc7557d037439: Status 404 returned error can't find the container with id 1969e6eec251f4d51169ccca9aba8b100ae835a3dcc2b9fac30cc7557d037439 Oct 31 22:01:48.022752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.022714 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" event={"ID":"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69","Type":"ContainerStarted","Data":"29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2"} Oct 31 22:01:48.022976 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.022769 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" event={"ID":"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69","Type":"ContainerStarted","Data":"1969e6eec251f4d51169ccca9aba8b100ae835a3dcc2b9fac30cc7557d037439"} Oct 31 22:01:48.023392 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.023370 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:48.069118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.069062 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" podStartSLOduration=1.069048127 podStartE2EDuration="1.069048127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:48.066995367 +0000 UTC m=+2748.192102599" watchObservedRunningTime="2026-10-31 22:01:48.069048127 +0000 UTC m=+2748.194155357" Oct 31 22:01:48.122029 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:48.121997 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-b06b04d8d7ad: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:48.122029 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:48.122023 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:48.122242 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:48.122080 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials podName:2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69 nodeName:}" failed. No retries permitted until 2026-10-31 22:01:48.622059236 +0000 UTC m=+2748.747166447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69") : secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:48.448329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.448051 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6"] Oct 31 22:01:48.453638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.453610 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.464351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.464320 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6"] Oct 31 22:01:48.469040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.469014 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:01:48.469259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.469211 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:01:48.531742 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.531694 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.531972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.531798 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xlgk\" (UniqueName: \"kubernetes.io/projected/2ac2b563-6516-498d-959a-6ba52742b4a1-kube-api-access-8xlgk\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.531972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.531907 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.633334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.633286 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.633518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.633477 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.633603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.633528 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xlgk\" (UniqueName: \"kubernetes.io/projected/2ac2b563-6516-498d-959a-6ba52742b4a1-kube-api-access-8xlgk\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.633603 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:48.633558 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-b06b04d8d7ad: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:48.633603 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:48.633580 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:48.633769 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:48.633640 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials podName:2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69 nodeName:}" failed. No retries permitted until 2026-10-31 22:01:49.633620987 +0000 UTC m=+2749.758728209 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69") : secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:48.634233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.634209 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.635827 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.635803 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.657031 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.657005 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xlgk\" (UniqueName: \"kubernetes.io/projected/2ac2b563-6516-498d-959a-6ba52742b4a1-kube-api-access-8xlgk\") pod \"maas-api-revoke-sub-498c255c-trkh6\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:48.783353 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:48.783313 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:49.082386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.082266 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz"] Oct 31 22:01:49.082664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.082646 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j"] Oct 31 22:01:49.135860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.132840 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.146892 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.146857 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j"] Oct 31 22:01:49.188731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.185500 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.188731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.185630 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcsbg\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-kube-api-access-jcsbg\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.188731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.185711 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.188731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.185972 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.287665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.287625 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.287856 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.287714 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcsbg\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-kube-api-access-jcsbg\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.287856 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.287763 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.287984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.287906 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.288474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.288444 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.288627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.288506 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.290121 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.290099 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.300054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.300032 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6"] Oct 31 22:01:49.300802 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:49.300778 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac2b563_6516_498d_959a_6ba52742b4a1.slice/crio-76d0877b4e9cdc15de03b8fa83bccf0676552e508ee10101b819d1c19bb36548 WatchSource:0}: Error finding container 76d0877b4e9cdc15de03b8fa83bccf0676552e508ee10101b819d1c19bb36548: Status 404 returned error can't find the container with id 76d0877b4e9cdc15de03b8fa83bccf0676552e508ee10101b819d1c19bb36548 Oct 31 22:01:49.311646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.311623 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcsbg\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-kube-api-access-jcsbg\") pod \"payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.495112 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.495082 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:49.693353 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:49.693317 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-b06b04d8d7ad: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:49.693353 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:49.693346 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:49.693551 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:49.693397 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials podName:2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69 nodeName:}" failed. No retries permitted until 2026-10-31 22:01:51.69338348 +0000 UTC m=+2751.818490690 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69") : secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:49.798507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:49.798477 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j"] Oct 31 22:01:49.798660 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:49.798637 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a4b057_e882_481d_86b3_0e5657c55834.slice/crio-5bede06bd18d11b2bc56f2079e69c445aa379b2e6a5299364cb5c554e1131f02 WatchSource:0}: Error finding container 5bede06bd18d11b2bc56f2079e69c445aa379b2e6a5299364cb5c554e1131f02: Status 404 returned error can't find the container with id 5bede06bd18d11b2bc56f2079e69c445aa379b2e6a5299364cb5c554e1131f02 Oct 31 22:01:50.147078 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.146979 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" event={"ID":"34a4b057-e882-481d-86b3-0e5657c55834","Type":"ContainerStarted","Data":"7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b"} Oct 31 22:01:50.147078 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.147040 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" event={"ID":"34a4b057-e882-481d-86b3-0e5657c55834","Type":"ContainerStarted","Data":"5bede06bd18d11b2bc56f2079e69c445aa379b2e6a5299364cb5c554e1131f02"} Oct 31 22:01:50.147335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.147093 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:01:50.149481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.149453 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" event={"ID":"2ac2b563-6516-498d-959a-6ba52742b4a1","Type":"ContainerStarted","Data":"d81cf0c018f7920d572c490161fdfb87028f5579b9ac488db45536546b114695"} Oct 31 22:01:50.149645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.149513 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" event={"ID":"2ac2b563-6516-498d-959a-6ba52742b4a1","Type":"ContainerStarted","Data":"76d0877b4e9cdc15de03b8fa83bccf0676552e508ee10101b819d1c19bb36548"} Oct 31 22:01:50.191450 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.191372 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" podStartSLOduration=1.191281777 podStartE2EDuration="1.191281777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:50.190926065 +0000 UTC m=+2750.316033313" watchObservedRunningTime="2026-10-31 22:01:50.191281777 +0000 UTC m=+2750.316389010" Oct 31 22:01:50.233075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.233017 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" podStartSLOduration=2.233001662 podStartE2EDuration="2.233001662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:01:50.232502065 +0000 UTC m=+2750.357609297" watchObservedRunningTime="2026-10-31 22:01:50.233001662 +0000 UTC m=+2750.358108893" Oct 31 22:01:50.966312 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:50.966266 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:01:51.159874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:51.159839 4294 generic.go:356] "Generic (PLEG): container finished" podID="2ac2b563-6516-498d-959a-6ba52742b4a1" containerID="d81cf0c018f7920d572c490161fdfb87028f5579b9ac488db45536546b114695" exitCode=0 Oct 31 22:01:51.160063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:51.159921 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" event={"ID":"2ac2b563-6516-498d-959a-6ba52742b4a1","Type":"ContainerDied","Data":"d81cf0c018f7920d572c490161fdfb87028f5579b9ac488db45536546b114695"} Oct 31 22:01:51.728795 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:51.728764 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-b06b04d8d7ad: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:51.728795 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:51.728788 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz: secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:51.729044 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:51.728840 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials podName:2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69 nodeName:}" failed. No retries permitted until 2026-10-31 22:01:55.728827356 +0000 UTC m=+2755.853934566 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69") : secret "agc-cred-b06b04d8d7ad" not found Oct 31 22:01:52.326341 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.326275 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:52.441606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.441526 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-openshift-ca\") pod \"2ac2b563-6516-498d-959a-6ba52742b4a1\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " Oct 31 22:01:52.441606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.441592 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac2b563-6516-498d-959a-6ba52742b4a1-kube-api-access-8xlgk\" (UniqueName: \"kubernetes.io/projected/2ac2b563-6516-498d-959a-6ba52742b4a1-kube-api-access-8xlgk\") pod \"2ac2b563-6516-498d-959a-6ba52742b4a1\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " Oct 31 22:01:52.442018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.441719 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-xks-ca\") pod \"2ac2b563-6516-498d-959a-6ba52742b4a1\" (UID: \"2ac2b563-6516-498d-959a-6ba52742b4a1\") " Oct 31 22:01:52.442018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.441947 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-openshift-ca" pod "2ac2b563-6516-498d-959a-6ba52742b4a1" (UID: "2ac2b563-6516-498d-959a-6ba52742b4a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:52.442265 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.442225 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:52.444280 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.444247 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-xks-ca" pod "2ac2b563-6516-498d-959a-6ba52742b4a1" (UID: "2ac2b563-6516-498d-959a-6ba52742b4a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:01:52.444404 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.444352 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac2b563-6516-498d-959a-6ba52742b4a1-kube-api-access-8xlgk" pod "2ac2b563-6516-498d-959a-6ba52742b4a1" (UID: "2ac2b563-6516-498d-959a-6ba52742b4a1"). InnerVolumeSpecName "kube-api-access-8xlgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:52.543703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.543667 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac2b563-6516-498d-959a-6ba52742b4a1-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:52.543703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:52.543696 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xlgk\" (UniqueName: \"kubernetes.io/projected/2ac2b563-6516-498d-959a-6ba52742b4a1-kube-api-access-8xlgk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:53.219217 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:53.219178 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" event={"ID":"2ac2b563-6516-498d-959a-6ba52742b4a1","Type":"ContainerDied","Data":"76d0877b4e9cdc15de03b8fa83bccf0676552e508ee10101b819d1c19bb36548"} Oct 31 22:01:53.219469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:53.219240 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76d0877b4e9cdc15de03b8fa83bccf0676552e508ee10101b819d1c19bb36548" Oct 31 22:01:53.219469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:53.219359 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6" Oct 31 22:01:55.150083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.150043 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" podUID="2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" containerName="payload-processing" containerID="cri-o://29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2" gracePeriod=25 Oct 31 22:01:55.421735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.421711 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:55.511680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.511647 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials\") pod \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " Oct 31 22:01:55.511852 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.511691 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-kube-api-access-tvq28\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-kube-api-access-tvq28\") pod \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " Oct 31 22:01:55.511927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.511908 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-plugins-config-volume\") pod \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " Oct 31 22:01:55.512011 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.511992 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-routing-overlay-volume\") pod \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\" (UID: \"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69\") " Oct 31 22:01:55.512406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.512378 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-plugins-config-volume" pod "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:55.512537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.512388 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-routing-overlay-volume" pod "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:01:55.512693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.512676 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:55.512693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.512694 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:55.514184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.514163 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-kube-api-access-tvq28" pod "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69"). InnerVolumeSpecName "kube-api-access-tvq28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:55.514263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.514212 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials" pod "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" (UID: "2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:01:55.613686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.613649 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:55.613686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:55.613682 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvq28\" (UniqueName: \"kubernetes.io/projected/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69-kube-api-access-tvq28\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:01:56.241316 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.241273 4294 generic.go:356] "Generic (PLEG): container finished" podID="2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" containerID="29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2" exitCode=0 Oct 31 22:01:56.241812 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.241334 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" Oct 31 22:01:56.241812 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.241363 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" event={"ID":"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69","Type":"ContainerDied","Data":"29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2"} Oct 31 22:01:56.241812 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.241440 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz" event={"ID":"2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69","Type":"ContainerDied","Data":"1969e6eec251f4d51169ccca9aba8b100ae835a3dcc2b9fac30cc7557d037439"} Oct 31 22:01:56.241812 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.241479 4294 scope.go:122] "RemoveContainer" containerID="29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2" Oct 31 22:01:56.254430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.253406 4294 scope.go:122] "RemoveContainer" containerID="29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2" Oct 31 22:01:56.254729 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:01:56.254701 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2\": container with ID starting with 29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2 not found: ID does not exist" containerID="29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2" Oct 31 22:01:56.254819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.254733 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2"} err="failed to get container status \"29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2\": rpc error: code = NotFound desc = could not find container \"29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2\": container with ID starting with 29ce16045eae2501324306f4f87b060d625b3d70163e291e52ecca4a6d06cdb2 not found: ID does not exist" Oct 31 22:01:56.294144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.294117 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz"] Oct 31 22:01:56.308226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:56.308198 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-66b4ffvbz"] Oct 31 22:01:57.985643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:57.985610 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" path="/var/lib/kubelet/pods/2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69/volumes" Oct 31 22:01:58.868241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.868212 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7"] Oct 31 22:01:58.868835 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.868821 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" containerName="payload-processing" Oct 31 22:01:58.868883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.868838 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" containerName="payload-processing" Oct 31 22:01:58.868883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.868878 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2ac2b563-6516-498d-959a-6ba52742b4a1" containerName="revoke-keys" Oct 31 22:01:58.868953 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.868884 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac2b563-6516-498d-959a-6ba52742b4a1" containerName="revoke-keys" Oct 31 22:01:58.868990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.868965 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac2b563-6516-498d-959a-6ba52742b4a1" containerName="revoke-keys" Oct 31 22:01:58.868990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.868978 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c488cdd-c7ae-4f68-9ab7-717fd2cf8f69" containerName="payload-processing" Oct 31 22:01:58.874101 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.874082 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:58.889850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.889828 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7"] Oct 31 22:01:58.897775 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.897751 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"agc-cred-ee3e6e124887\"" Oct 31 22:01:58.897953 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.897842 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"payload-processing-external-model-e2e--6e09a196-dockercfg-tqvxc\"" Oct 31 22:01:58.898080 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.897867 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"routing-overlay\"" Oct 31 22:01:58.898080 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.897900 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"openshift-service-ca.crt\"" Oct 31 22:01:58.903266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.903245 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4275b9\"" Oct 31 22:01:58.903417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.903266 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"kube-root-ca.crt\"" Oct 31 22:01:58.975145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.975021 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.002980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:58.975412 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.002980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.001531 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhm62\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-kube-api-access-bhm62\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.002980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.001711 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.103541 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.103503 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhm62\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-kube-api-access-bhm62\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.103738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.103623 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.103738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.103715 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.103819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.103751 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.104951 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.104925 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.105692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.105665 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.118171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.118142 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.127852 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.127792 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhm62\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-kube-api-access-bhm62\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.187687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.187658 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:01:59.490246 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:01:59.490220 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7"] Oct 31 22:01:59.490873 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:01:59.490850 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45aed7c7_0f6c_4055_94f2_74bbd89d5d4b.slice/crio-3a25eb425de8571507e92fcc640698ebc96b7a6889dd17d93a5c229db6aace33 WatchSource:0}: Error finding container 3a25eb425de8571507e92fcc640698ebc96b7a6889dd17d93a5c229db6aace33: Status 404 returned error can't find the container with id 3a25eb425de8571507e92fcc640698ebc96b7a6889dd17d93a5c229db6aace33 Oct 31 22:02:00.274330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:00.274267 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" event={"ID":"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b","Type":"ContainerStarted","Data":"2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930"} Oct 31 22:02:00.274755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:00.274354 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" event={"ID":"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b","Type":"ContainerStarted","Data":"3a25eb425de8571507e92fcc640698ebc96b7a6889dd17d93a5c229db6aace33"} Oct 31 22:02:00.274755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:00.274397 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:02:00.321569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:00.321523 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" podStartSLOduration=2.32150917 podStartE2EDuration="2.32150917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:00.317519573 +0000 UTC m=+2760.442626829" watchObservedRunningTime="2026-10-31 22:02:00.32150917 +0000 UTC m=+2760.446616400" Oct 31 22:02:01.163486 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:01.163456 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:02:01.234592 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:01.234559 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-7cc44fcedad2: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:01.234592 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:01.234588 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:01.234842 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:01.234644 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials podName:34a4b057-e882-481d-86b3-0e5657c55834 nodeName:}" failed. No retries permitted until 2026-10-31 22:02:01.734626667 +0000 UTC m=+2761.859733896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" (UID: "34a4b057-e882-481d-86b3-0e5657c55834") : secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:01.311772 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:01.311740 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:02:01.741145 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:01.741114 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-7cc44fcedad2: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:01.741145 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:01.741139 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:01.741440 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:01.741189 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials podName:34a4b057-e882-481d-86b3-0e5657c55834 nodeName:}" failed. No retries permitted until 2026-10-31 22:02:02.74117521 +0000 UTC m=+2762.866282419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" (UID: "34a4b057-e882-481d-86b3-0e5657c55834") : secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:02.608459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:02.608430 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j"] Oct 31 22:02:02.756952 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:02.756916 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-7cc44fcedad2: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:02.756952 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:02.756944 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:02.757178 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:02.756996 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials podName:34a4b057-e882-481d-86b3-0e5657c55834 nodeName:}" failed. No retries permitted until 2026-10-31 22:02:04.756981398 +0000 UTC m=+2764.882088608 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" (UID: "34a4b057-e882-481d-86b3-0e5657c55834") : secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:04.802002 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:04.801971 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-fac043/agc-cred-7cc44fcedad2: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:04.802002 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:04.801998 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j: secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:04.802568 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:04.802064 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials podName:34a4b057-e882-481d-86b3-0e5657c55834 nodeName:}" failed. No retries permitted until 2026-10-31 22:02:08.802044961 +0000 UTC m=+2768.927152177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" (UID: "34a4b057-e882-481d-86b3-0e5657c55834") : secret "agc-cred-7cc44fcedad2" not found Oct 31 22:02:05.279473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:05.279438 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7"] Oct 31 22:02:05.282790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:05.282763 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:02:06.361036 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.361003 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j"] Oct 31 22:02:06.366485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.366468 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.384768 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.384748 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j"] Oct 31 22:02:06.391955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.391933 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:02:06.392094 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.392017 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:02:06.446255 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.446226 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.446420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.446392 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pw64\" (UniqueName: \"kubernetes.io/projected/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-kube-api-access-5pw64\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.446468 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.446436 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.547985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.547948 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pw64\" (UniqueName: \"kubernetes.io/projected/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-kube-api-access-5pw64\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.548136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.548021 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.548136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.548117 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.548758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.548732 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.550364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.550342 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.573609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.573576 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pw64\" (UniqueName: \"kubernetes.io/projected/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-kube-api-access-5pw64\") pod \"maas-api-revoke-sub-a05f4ee3-7j97j\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.680390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.680294 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:06.982266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:06.982232 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j"] Oct 31 22:02:06.990942 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:02:06.990914 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f16b1c1_5fd5_4e3e_b9bc_9703585d0492.slice/crio-10fa0835b2e8e04bef5aa7e77221545e34b3a073d83721593605313245d27051 WatchSource:0}: Error finding container 10fa0835b2e8e04bef5aa7e77221545e34b3a073d83721593605313245d27051: Status 404 returned error can't find the container with id 10fa0835b2e8e04bef5aa7e77221545e34b3a073d83721593605313245d27051 Oct 31 22:02:07.333353 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.333313 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" event={"ID":"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492","Type":"ContainerStarted","Data":"0c81e5cc345da57662f8f2c685489a662e2ee93f232ced9479409aaf3f338b55"} Oct 31 22:02:07.333526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.333378 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" event={"ID":"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492","Type":"ContainerStarted","Data":"10fa0835b2e8e04bef5aa7e77221545e34b3a073d83721593605313245d27051"} Oct 31 22:02:07.373352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.373288 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" podStartSLOduration=1.3732731089999999 podStartE2EDuration="1.373273109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:07.372275787 +0000 UTC m=+2767.497383017" watchObservedRunningTime="2026-10-31 22:02:07.373273109 +0000 UTC m=+2767.498380340" Oct 31 22:02:07.609052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.608967 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" podUID="34a4b057-e882-481d-86b3-0e5657c55834" containerName="payload-processing" containerID="cri-o://7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b" gracePeriod=25 Oct 31 22:02:07.870925 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.870868 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:02:07.969874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.969837 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-plugins-config-volume\") pod \"34a4b057-e882-481d-86b3-0e5657c55834\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " Oct 31 22:02:07.970117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.969964 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials\") pod \"34a4b057-e882-481d-86b3-0e5657c55834\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " Oct 31 22:02:07.970117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.970059 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-routing-overlay-volume\") pod \"34a4b057-e882-481d-86b3-0e5657c55834\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " Oct 31 22:02:07.970253 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.970236 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-kube-api-access-jcsbg\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-kube-api-access-jcsbg\") pod \"34a4b057-e882-481d-86b3-0e5657c55834\" (UID: \"34a4b057-e882-481d-86b3-0e5657c55834\") " Oct 31 22:02:07.982977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.980546 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-plugins-config-volume" pod "34a4b057-e882-481d-86b3-0e5657c55834" (UID: "34a4b057-e882-481d-86b3-0e5657c55834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:02:07.982977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.981712 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-routing-overlay-volume" pod "34a4b057-e882-481d-86b3-0e5657c55834" (UID: "34a4b057-e882-481d-86b3-0e5657c55834"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:02:07.982977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.982776 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-kube-api-access-jcsbg" pod "34a4b057-e882-481d-86b3-0e5657c55834" (UID: "34a4b057-e882-481d-86b3-0e5657c55834"). InnerVolumeSpecName "kube-api-access-jcsbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:02:07.988831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:07.988799 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials" pod "34a4b057-e882-481d-86b3-0e5657c55834" (UID: "34a4b057-e882-481d-86b3-0e5657c55834"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:02:08.073659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.073627 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcsbg\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-kube-api-access-jcsbg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:08.073659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.073655 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:08.073659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.073665 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/34a4b057-e882-481d-86b3-0e5657c55834-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:08.073888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.073675 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/34a4b057-e882-481d-86b3-0e5657c55834-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:08.342262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.342228 4294 generic.go:356] "Generic (PLEG): container finished" podID="8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" containerID="0c81e5cc345da57662f8f2c685489a662e2ee93f232ced9479409aaf3f338b55" exitCode=0 Oct 31 22:02:08.342466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.342317 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" event={"ID":"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492","Type":"ContainerDied","Data":"0c81e5cc345da57662f8f2c685489a662e2ee93f232ced9479409aaf3f338b55"} Oct 31 22:02:08.343713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.343690 4294 generic.go:356] "Generic (PLEG): container finished" podID="34a4b057-e882-481d-86b3-0e5657c55834" containerID="7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b" exitCode=0 Oct 31 22:02:08.343818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.343727 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" Oct 31 22:02:08.343818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.343731 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" event={"ID":"34a4b057-e882-481d-86b3-0e5657c55834","Type":"ContainerDied","Data":"7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b"} Oct 31 22:02:08.343818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.343764 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j" event={"ID":"34a4b057-e882-481d-86b3-0e5657c55834","Type":"ContainerDied","Data":"5bede06bd18d11b2bc56f2079e69c445aa379b2e6a5299364cb5c554e1131f02"} Oct 31 22:02:08.343818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.343790 4294 scope.go:122] "RemoveContainer" containerID="7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b" Oct 31 22:02:08.353687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.353666 4294 scope.go:122] "RemoveContainer" containerID="7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b" Oct 31 22:02:08.353978 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:08.353949 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b\": container with ID starting with 7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b not found: ID does not exist" containerID="7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b" Oct 31 22:02:08.354081 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.353976 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b"} err="failed to get container status \"7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b\": rpc error: code = NotFound desc = could not find container \"7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b\": container with ID starting with 7a14503fd2534f3223fda734e613971bc1a6f177b9757d296761262154e5103b not found: ID does not exist" Oct 31 22:02:08.422902 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.422770 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j"] Oct 31 22:02:08.428749 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:08.428721 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-fac043/payload-processing-external-model-e2e-extmod-a-fac043-5bf6gcr2j"] Oct 31 22:02:09.496351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.496327 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:09.596420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.596386 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-kube-api-access-5pw64\" (UniqueName: \"kubernetes.io/projected/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-kube-api-access-5pw64\") pod \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " Oct 31 22:02:09.596593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.596458 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-xks-ca\") pod \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " Oct 31 22:02:09.596593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.596565 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-openshift-ca\") pod \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\" (UID: \"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492\") " Oct 31 22:02:09.596929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.596906 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-openshift-ca" pod "8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" (UID: "8f16b1c1-5fd5-4e3e-b9bc-9703585d0492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:02:09.598561 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.598534 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-xks-ca" pod "8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" (UID: "8f16b1c1-5fd5-4e3e-b9bc-9703585d0492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:02:09.598661 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.598594 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-kube-api-access-5pw64" pod "8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" (UID: "8f16b1c1-5fd5-4e3e-b9bc-9703585d0492"). InnerVolumeSpecName "kube-api-access-5pw64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:02:09.698205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.698174 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pw64\" (UniqueName: \"kubernetes.io/projected/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-kube-api-access-5pw64\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:09.698205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.698199 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:09.698205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.698209 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:09.986169 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:09.986141 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34a4b057-e882-481d-86b3-0e5657c55834" path="/var/lib/kubelet/pods/34a4b057-e882-481d-86b3-0e5657c55834/volumes" Oct 31 22:02:10.279930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.279832 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" podUID="45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" containerName="payload-processing" containerID="cri-o://2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930" gracePeriod=25 Oct 31 22:02:10.360790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.360755 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" Oct 31 22:02:10.360943 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.360786 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j" event={"ID":"8f16b1c1-5fd5-4e3e-b9bc-9703585d0492","Type":"ContainerDied","Data":"10fa0835b2e8e04bef5aa7e77221545e34b3a073d83721593605313245d27051"} Oct 31 22:02:10.360943 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.360838 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10fa0835b2e8e04bef5aa7e77221545e34b3a073d83721593605313245d27051" Oct 31 22:02:10.559491 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.559466 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:02:10.713047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713008 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-provider-credentials\") pod \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " Oct 31 22:02:10.713247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713062 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-routing-overlay-volume\") pod \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " Oct 31 22:02:10.713247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713215 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-plugins-config-volume\") pod \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " Oct 31 22:02:10.713399 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713275 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-kube-api-access-bhm62\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-kube-api-access-bhm62\") pod \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\" (UID: \"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b\") " Oct 31 22:02:10.713559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713530 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-routing-overlay-volume" pod "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" (UID: "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:02:10.713636 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713616 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-plugins-config-volume" pod "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" (UID: "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:02:10.713869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713847 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:10.713869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.713865 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:10.715496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.715470 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-provider-credentials" pod "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" (UID: "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:02:10.715496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.715482 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-kube-api-access-bhm62" pod "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" (UID: "45aed7c7-0f6c-4055-94f2-74bbd89d5d4b"). InnerVolumeSpecName "kube-api-access-bhm62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:02:10.815203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.815108 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:10.815203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:10.815139 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhm62\" (UniqueName: \"kubernetes.io/projected/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b-kube-api-access-bhm62\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:11.369490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.369455 4294 generic.go:356] "Generic (PLEG): container finished" podID="45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" containerID="2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930" exitCode=0 Oct 31 22:02:11.369682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.369522 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" Oct 31 22:02:11.369682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.369534 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" event={"ID":"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b","Type":"ContainerDied","Data":"2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930"} Oct 31 22:02:11.369682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.369589 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7" event={"ID":"45aed7c7-0f6c-4055-94f2-74bbd89d5d4b","Type":"ContainerDied","Data":"3a25eb425de8571507e92fcc640698ebc96b7a6889dd17d93a5c229db6aace33"} Oct 31 22:02:11.369682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.369613 4294 scope.go:122] "RemoveContainer" containerID="2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930" Oct 31 22:02:11.380515 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.380501 4294 scope.go:122] "RemoveContainer" containerID="2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930" Oct 31 22:02:11.380791 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:11.380772 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930\": container with ID starting with 2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930 not found: ID does not exist" containerID="2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930" Oct 31 22:02:11.380845 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.380800 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930"} err="failed to get container status \"2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930\": rpc error: code = NotFound desc = could not find container \"2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930\": container with ID starting with 2a33c122474cfcdf56d93c27e37d6c6969d8dca250fa4760b603d15934379930 not found: ID does not exist" Oct 31 22:02:11.423012 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.422978 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7"] Oct 31 22:02:11.434695 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.434672 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-84d8ftjg7"] Oct 31 22:02:11.985860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:11.985825 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" path="/var/lib/kubelet/pods/45aed7c7-0f6c-4055-94f2-74bbd89d5d4b/volumes" Oct 31 22:02:17.144159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144116 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r"] Oct 31 22:02:17.144760 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144745 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" containerName="revoke-keys" Oct 31 22:02:17.144813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144761 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" containerName="revoke-keys" Oct 31 22:02:17.144813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144781 4294 cpu_manager.go:414] "containerMap: removing container" podUID="45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" containerName="payload-processing" Oct 31 22:02:17.144813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144786 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" containerName="payload-processing" Oct 31 22:02:17.144813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144800 4294 cpu_manager.go:414] "containerMap: removing container" podUID="34a4b057-e882-481d-86b3-0e5657c55834" containerName="payload-processing" Oct 31 22:02:17.144813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144806 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34a4b057-e882-481d-86b3-0e5657c55834" containerName="payload-processing" Oct 31 22:02:17.144967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144926 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="34a4b057-e882-481d-86b3-0e5657c55834" containerName="payload-processing" Oct 31 22:02:17.144967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144942 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" containerName="revoke-keys" Oct 31 22:02:17.144967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.144950 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="45aed7c7-0f6c-4055-94f2-74bbd89d5d4b" containerName="payload-processing" Oct 31 22:02:17.148971 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.148947 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.168524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.168499 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r"] Oct 31 22:02:17.175352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.174558 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a041b65867da\"" Oct 31 22:02:17.175352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.174580 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 22:02:17.175352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.174640 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 22:02:17.175352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.174647 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 22:02:17.175352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.174723 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 22:02:17.175352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.174762 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-31f7549f762e\"" Oct 31 22:02:17.175860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.175843 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-tgxz6\"" Oct 31 22:02:17.215562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.215535 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-provider-credentials\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.215715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.215602 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcv8\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-kube-api-access-tpcv8\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.215794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.215720 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-routing-overlay-volume\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.215893 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.215875 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-plugins-config-volume\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.316955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.316907 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-provider-credentials\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.317189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.317093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcv8\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-kube-api-access-tpcv8\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.317189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.317179 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-routing-overlay-volume\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.317363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.317335 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-plugins-config-volume\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.318164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.318145 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-plugins-config-volume\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.318238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.318188 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-routing-overlay-volume\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.319992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.319969 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-provider-credentials\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.340445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.340422 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcv8\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-kube-api-access-tpcv8\") pod \"payload-processing-external-model-67ddf5bb68-5ss2r\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.465999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.465921 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:17.780765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:17.780737 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r"] Oct 31 22:02:17.781148 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:02:17.781124 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b487b2_55ee_40db_a33a_f22ad66c660c.slice/crio-11cdc26d72c04f571e058b6187a85db7c67e93005afe61280e6ce5e6de90109d WatchSource:0}: Error finding container 11cdc26d72c04f571e058b6187a85db7c67e93005afe61280e6ce5e6de90109d: Status 404 returned error can't find the container with id 11cdc26d72c04f571e058b6187a85db7c67e93005afe61280e6ce5e6de90109d Oct 31 22:02:18.421263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:18.421224 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" event={"ID":"84b487b2-55ee-40db-a33a-f22ad66c660c","Type":"ContainerStarted","Data":"963d5adbfd2b68f60434838682c7b3796634a3e96b43664088053d26e7519191"} Oct 31 22:02:18.421709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:18.421278 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" event={"ID":"84b487b2-55ee-40db-a33a-f22ad66c660c","Type":"ContainerStarted","Data":"11cdc26d72c04f571e058b6187a85db7c67e93005afe61280e6ce5e6de90109d"} Oct 31 22:02:18.421709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:18.421377 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:18.466752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:18.466703 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" podStartSLOduration=1.466689468 podStartE2EDuration="1.466689468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:18.46272024 +0000 UTC m=+2778.587827589" watchObservedRunningTime="2026-10-31 22:02:18.466689468 +0000 UTC m=+2778.591796698" Oct 31 22:02:29.431681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:29.431648 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:02:31.311144 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:02:31.311109 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:02:33.633959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.633925 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs"] Oct 31 22:02:33.638387 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.638365 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.657283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.657258 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs"] Oct 31 22:02:33.665348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.664684 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"e2e-cmp-legacy-1a22be-provider-cred\"" Oct 31 22:02:33.665348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.664785 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"openshift-service-ca.crt\"" Oct 31 22:02:33.665348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.664807 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"routing-overlay\"" Oct 31 22:02:33.665348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.664837 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4275b9\"" Oct 31 22:02:33.665348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.664697 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"payload-processing-external-model-e2e--6e09a196-dockercfg-v6zr6\"" Oct 31 22:02:33.665348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.665118 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"kube-root-ca.crt\"" Oct 31 22:02:33.783602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.783044 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76746\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-kube-api-access-76746\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.783602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.783250 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.783602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.783322 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.783602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.783353 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.885416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.885327 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76746\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-kube-api-access-76746\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.885582 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.885505 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.885582 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.885547 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.885582 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.885574 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.886281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.886257 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.886494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.886475 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.888414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.888392 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.908734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.908709 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76746\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-kube-api-access-76746\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:33.953005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:33.952983 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:34.258680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:34.258641 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs"] Oct 31 22:02:34.259908 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:02:34.259873 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba74cf99_5fb7_4c8e_8f5e_737afa143a09.slice/crio-495a17b62ee228184378a2d175efe9a32ce4826505e1130fb16578e8ca8a1c93 WatchSource:0}: Error finding container 495a17b62ee228184378a2d175efe9a32ce4826505e1130fb16578e8ca8a1c93: Status 404 returned error can't find the container with id 495a17b62ee228184378a2d175efe9a32ce4826505e1130fb16578e8ca8a1c93 Oct 31 22:02:34.594172 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:34.594087 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" event={"ID":"ba74cf99-5fb7-4c8e-8f5e-737afa143a09","Type":"ContainerStarted","Data":"4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915"} Oct 31 22:02:34.594172 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:34.594135 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" event={"ID":"ba74cf99-5fb7-4c8e-8f5e-737afa143a09","Type":"ContainerStarted","Data":"495a17b62ee228184378a2d175efe9a32ce4826505e1130fb16578e8ca8a1c93"} Oct 31 22:02:34.594172 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:34.594163 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:34.638125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:34.638055 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" podStartSLOduration=1.6380389229999999 podStartE2EDuration="1.638038923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:34.635500467 +0000 UTC m=+2794.760607697" watchObservedRunningTime="2026-10-31 22:02:34.638038923 +0000 UTC m=+2794.763146155" Oct 31 22:02:45.605942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:45.605905 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:02:55.331949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.331915 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875"] Oct 31 22:02:55.337599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.337574 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.354551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.354519 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875"] Oct 31 22:02:55.360974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.360950 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:02:55.361156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.361064 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:02:55.506607 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.506567 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx5q8\" (UniqueName: \"kubernetes.io/projected/6db49cf9-781a-4b25-8dae-56d00c4352f7-kube-api-access-bx5q8\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.506803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.506632 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.506803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.506721 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.608083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.608000 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx5q8\" (UniqueName: \"kubernetes.io/projected/6db49cf9-781a-4b25-8dae-56d00c4352f7-kube-api-access-bx5q8\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.608083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.608061 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.608290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.608093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.608869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.608847 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.610514 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.610490 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.631239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.631212 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx5q8\" (UniqueName: \"kubernetes.io/projected/6db49cf9-781a-4b25-8dae-56d00c4352f7-kube-api-access-bx5q8\") pod \"maas-api-revoke-sub-dae194f7-qw875\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.654466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.654437 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:55.966342 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:55.963821 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875"] Oct 31 22:02:55.966342 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:02:55.964861 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6db49cf9_781a_4b25_8dae_56d00c4352f7.slice/crio-812fbd2c6630c8ea1c55882f85f9f0ea29149ef9c098c691e00b449514ee3cf8 WatchSource:0}: Error finding container 812fbd2c6630c8ea1c55882f85f9f0ea29149ef9c098c691e00b449514ee3cf8: Status 404 returned error can't find the container with id 812fbd2c6630c8ea1c55882f85f9f0ea29149ef9c098c691e00b449514ee3cf8 Oct 31 22:02:56.383624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.383591 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf"] Oct 31 22:02:56.388101 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.388080 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420065 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420029 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b584485-3ab6-4857-8701-7ec674cabe77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420090 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b584485-3ab6-4857-8701-7ec674cabe77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420229 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420617 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420590 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420739 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420803 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420877 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420853 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.420934 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.420899 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.421118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.421103 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plww\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-kube-api-access-5plww\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.431339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.428628 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf"] Oct 31 22:02:56.522267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522222 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b584485-3ab6-4857-8701-7ec674cabe77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522278 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522351 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522461 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522512 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522554 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522610 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522705 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plww\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-kube-api-access-5plww\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522806 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.522867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522825 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b584485-3ab6-4857-8701-7ec674cabe77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.523022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522948 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.523022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.522970 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.523199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.523021 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.523199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.523135 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b584485-3ab6-4857-8701-7ec674cabe77-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.524741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.524722 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.525055 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.525038 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b584485-3ab6-4857-8701-7ec674cabe77-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.545552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.545526 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.546444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.546428 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plww\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-kube-api-access-5plww\") pod \"maas-default-gateway-openshift-default-58847f9fc5-5drlf\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.702435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.702403 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:56.811352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.811244 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" event={"ID":"6db49cf9-781a-4b25-8dae-56d00c4352f7","Type":"ContainerStarted","Data":"7857c5f462c369caf8183119bc01ac11622a6a7684ca4043071be553dd27b49c"} Oct 31 22:02:56.811352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.811331 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" event={"ID":"6db49cf9-781a-4b25-8dae-56d00c4352f7","Type":"ContainerStarted","Data":"812fbd2c6630c8ea1c55882f85f9f0ea29149ef9c098c691e00b449514ee3cf8"} Oct 31 22:02:56.855328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:56.855270 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" podStartSLOduration=1.855256604 podStartE2EDuration="1.855256604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:56.851760414 +0000 UTC m=+2816.976867645" watchObservedRunningTime="2026-10-31 22:02:56.855256604 +0000 UTC m=+2816.980363834" Oct 31 22:02:57.038588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.038564 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf"] Oct 31 22:02:57.039213 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:02:57.039184 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b584485_3ab6_4857_8701_7ec674cabe77.slice/crio-949a9012a3e5d5ba6a6511f86e1c853febd9bafa2e8fd73632574af9cdce56a8 WatchSource:0}: Error finding container 949a9012a3e5d5ba6a6511f86e1c853febd9bafa2e8fd73632574af9cdce56a8: Status 404 returned error can't find the container with id 949a9012a3e5d5ba6a6511f86e1c853febd9bafa2e8fd73632574af9cdce56a8 Oct 31 22:02:57.041988 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.041953 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:02:57.042087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.042029 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:02:57.042087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.042072 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:02:57.822831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.822792 4294 generic.go:356] "Generic (PLEG): container finished" podID="6db49cf9-781a-4b25-8dae-56d00c4352f7" containerID="7857c5f462c369caf8183119bc01ac11622a6a7684ca4043071be553dd27b49c" exitCode=0 Oct 31 22:02:57.823362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.822874 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" event={"ID":"6db49cf9-781a-4b25-8dae-56d00c4352f7","Type":"ContainerDied","Data":"7857c5f462c369caf8183119bc01ac11622a6a7684ca4043071be553dd27b49c"} Oct 31 22:02:57.825060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.825032 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" event={"ID":"7b584485-3ab6-4857-8701-7ec674cabe77","Type":"ContainerStarted","Data":"9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d"} Oct 31 22:02:57.825210 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.825088 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" event={"ID":"7b584485-3ab6-4857-8701-7ec674cabe77","Type":"ContainerStarted","Data":"949a9012a3e5d5ba6a6511f86e1c853febd9bafa2e8fd73632574af9cdce56a8"} Oct 31 22:02:57.915864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:57.915814 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podStartSLOduration=1.91580034 podStartE2EDuration="1.91580034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:02:57.911700065 +0000 UTC m=+2818.036807308" watchObservedRunningTime="2026-10-31 22:02:57.91580034 +0000 UTC m=+2818.040907571" Oct 31 22:02:58.703253 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:58.703217 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:02:58.704768 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:58.704741 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 31 22:02:58.704899 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:58.704787 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 31 22:02:59.032464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.032434 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:59.064828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.064801 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-openshift-ca\") pod \"6db49cf9-781a-4b25-8dae-56d00c4352f7\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " Oct 31 22:02:59.064952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.064904 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-xks-ca\") pod \"6db49cf9-781a-4b25-8dae-56d00c4352f7\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " Oct 31 22:02:59.064994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.064968 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6db49cf9-781a-4b25-8dae-56d00c4352f7-kube-api-access-bx5q8\" (UniqueName: \"kubernetes.io/projected/6db49cf9-781a-4b25-8dae-56d00c4352f7-kube-api-access-bx5q8\") pod \"6db49cf9-781a-4b25-8dae-56d00c4352f7\" (UID: \"6db49cf9-781a-4b25-8dae-56d00c4352f7\") " Oct 31 22:02:59.065201 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.065175 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-openshift-ca" pod "6db49cf9-781a-4b25-8dae-56d00c4352f7" (UID: "6db49cf9-781a-4b25-8dae-56d00c4352f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:02:59.065565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.065541 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:59.067009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.066987 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-xks-ca" pod "6db49cf9-781a-4b25-8dae-56d00c4352f7" (UID: "6db49cf9-781a-4b25-8dae-56d00c4352f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:02:59.067133 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.067114 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6db49cf9-781a-4b25-8dae-56d00c4352f7-kube-api-access-bx5q8" pod "6db49cf9-781a-4b25-8dae-56d00c4352f7" (UID: "6db49cf9-781a-4b25-8dae-56d00c4352f7"). InnerVolumeSpecName "kube-api-access-bx5q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:02:59.167050 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.167020 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6db49cf9-781a-4b25-8dae-56d00c4352f7-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:59.167050 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.167047 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx5q8\" (UniqueName: \"kubernetes.io/projected/6db49cf9-781a-4b25-8dae-56d00c4352f7-kube-api-access-bx5q8\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:02:59.703040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.703003 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 31 22:02:59.703225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.703060 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 31 22:02:59.841324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.841268 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" Oct 31 22:02:59.841324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.841295 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875" event={"ID":"6db49cf9-781a-4b25-8dae-56d00c4352f7","Type":"ContainerDied","Data":"812fbd2c6630c8ea1c55882f85f9f0ea29149ef9c098c691e00b449514ee3cf8"} Oct 31 22:02:59.841545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:02:59.841369 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="812fbd2c6630c8ea1c55882f85f9f0ea29149ef9c098c691e00b449514ee3cf8" Oct 31 22:03:00.703395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:00.703357 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 31 22:03:00.703822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:00.703416 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 31 22:03:01.325362 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:01.325326 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:03:01.702830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:01.702748 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 31 22:03:01.702830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:01.702801 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 31 22:03:02.610181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:02.610137 4294 scope.go:122] "RemoveContainer" containerID="d34d6190f336ffebe6de7e36e2a8655f381e2e996ff4f795c032b6ea113b788d" Oct 31 22:03:02.703526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:02.703492 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 31 22:03:02.703710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:02.703543 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 31 22:03:03.703728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:03.703694 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 31 22:03:03.704141 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:03.703748 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 31 22:03:04.702987 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:04.702949 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" start-of-body= Oct 31 22:03:04.703180 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:04.703002 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": dial tcp 10.134.0.255:15021: connect: connection refused" Oct 31 22:03:05.708070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:05.708041 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:03:05.708564 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:05.708550 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:03:05.709263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:05.709241 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:03:06.948364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:06.948322 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf"] Oct 31 22:03:11.451781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.451750 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz"] Oct 31 22:03:11.452381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.452367 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6db49cf9-781a-4b25-8dae-56d00c4352f7" containerName="revoke-keys" Oct 31 22:03:11.452443 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.452382 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6db49cf9-781a-4b25-8dae-56d00c4352f7" containerName="revoke-keys" Oct 31 22:03:11.452519 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.452510 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6db49cf9-781a-4b25-8dae-56d00c4352f7" containerName="revoke-keys" Oct 31 22:03:11.457475 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.457454 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.480896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.480867 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz"] Oct 31 22:03:11.582402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582366 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582411 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582433 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582576 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582616 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582644 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582769 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582828 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.582928 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.582902 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gls92\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-kube-api-access-gls92\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684331 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684544 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684401 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684544 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684439 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684544 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684515 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gls92\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-kube-api-access-gls92\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684676 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684660 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684706 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684743 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684837 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684808 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684845 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.684900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684847 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.685020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.684898 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.685145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.685118 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.685237 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.685142 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.685525 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.685506 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.686825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.686805 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.686933 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.686859 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.707588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.707537 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.707684 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.707632 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gls92\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-kube-api-access-gls92\") pod \"maas-default-gateway-openshift-default-58847f9fc5-kf6vz\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.780272 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.780245 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:11.949273 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:11.949212 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" podUID="00138ef9-1ba8-4578-b64a-72d00ee8bcdd" containerName="payload-processing" containerID="cri-o://b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8" gracePeriod=25 Oct 31 22:03:12.104222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.104189 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz"] Oct 31 22:03:12.105989 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:03:12.105957 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919aad89_5a32_4eb8_bb0a_dfaf842ac2db.slice/crio-51c5cdf20376836e5373d5c4116bbf8c32d5a759e4cd4783d58fb43a591e6c14 WatchSource:0}: Error finding container 51c5cdf20376836e5373d5c4116bbf8c32d5a759e4cd4783d58fb43a591e6c14: Status 404 returned error can't find the container with id 51c5cdf20376836e5373d5c4116bbf8c32d5a759e4cd4783d58fb43a591e6c14 Oct 31 22:03:12.108748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.108711 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:03:12.108832 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.108798 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:03:12.108888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.108836 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:03:12.254234 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.254204 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:03:12.293756 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.293727 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-provider-credentials\") pod \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " Oct 31 22:03:12.293957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.293797 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-plugins-config-volume\") pod \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " Oct 31 22:03:12.293957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.293830 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-kube-api-access-xb2db\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-kube-api-access-xb2db\") pod \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " Oct 31 22:03:12.293957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.293939 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-routing-overlay-volume\") pod \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\" (UID: \"00138ef9-1ba8-4578-b64a-72d00ee8bcdd\") " Oct 31 22:03:12.294667 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.294625 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-routing-overlay-volume" pod "00138ef9-1ba8-4578-b64a-72d00ee8bcdd" (UID: "00138ef9-1ba8-4578-b64a-72d00ee8bcdd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:12.295062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.295035 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-plugins-config-volume" pod "00138ef9-1ba8-4578-b64a-72d00ee8bcdd" (UID: "00138ef9-1ba8-4578-b64a-72d00ee8bcdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:12.297356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.296785 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-kube-api-access-xb2db" pod "00138ef9-1ba8-4578-b64a-72d00ee8bcdd" (UID: "00138ef9-1ba8-4578-b64a-72d00ee8bcdd"). InnerVolumeSpecName "kube-api-access-xb2db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:12.297356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.296912 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-provider-credentials" pod "00138ef9-1ba8-4578-b64a-72d00ee8bcdd" (UID: "00138ef9-1ba8-4578-b64a-72d00ee8bcdd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:12.396240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.396182 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:12.396240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.396227 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:12.396240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.396237 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb2db\" (UniqueName: \"kubernetes.io/projected/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-kube-api-access-xb2db\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:12.396240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.396246 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/00138ef9-1ba8-4578-b64a-72d00ee8bcdd-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:12.965230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.965134 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" event={"ID":"919aad89-5a32-4eb8-bb0a-dfaf842ac2db","Type":"ContainerStarted","Data":"5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38"} Oct 31 22:03:12.965230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.965219 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" event={"ID":"919aad89-5a32-4eb8-bb0a-dfaf842ac2db","Type":"ContainerStarted","Data":"51c5cdf20376836e5373d5c4116bbf8c32d5a759e4cd4783d58fb43a591e6c14"} Oct 31 22:03:12.966757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.966727 4294 generic.go:356] "Generic (PLEG): container finished" podID="00138ef9-1ba8-4578-b64a-72d00ee8bcdd" containerID="b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8" exitCode=0 Oct 31 22:03:12.966887 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.966794 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" Oct 31 22:03:12.966887 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.966817 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" event={"ID":"00138ef9-1ba8-4578-b64a-72d00ee8bcdd","Type":"ContainerDied","Data":"b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8"} Oct 31 22:03:12.966887 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.966874 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf" event={"ID":"00138ef9-1ba8-4578-b64a-72d00ee8bcdd","Type":"ContainerDied","Data":"64fd127e6e9a7b6bed9de1a934973c32caf07536bc930a3858aa8aa32f9b54ca"} Oct 31 22:03:12.967026 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:12.966905 4294 scope.go:122] "RemoveContainer" containerID="b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8" Oct 31 22:03:13.003631 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.003601 4294 scope.go:122] "RemoveContainer" containerID="b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8" Oct 31 22:03:13.004006 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:13.003979 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8\": container with ID starting with b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8 not found: ID does not exist" containerID="b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8" Oct 31 22:03:13.004107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.004015 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8"} err="failed to get container status \"b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8\": rpc error: code = NotFound desc = could not find container \"b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8\": container with ID starting with b68a15d996a5d27fc5f7231dea14f3715317f28856b19ff329857a37eaf3c2c8 not found: ID does not exist" Oct 31 22:03:13.023277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.023235 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podStartSLOduration=2.023223066 podStartE2EDuration="2.023223066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:03:13.022904957 +0000 UTC m=+2833.148012201" watchObservedRunningTime="2026-10-31 22:03:13.023223066 +0000 UTC m=+2833.148330296" Oct 31 22:03:13.061639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.061606 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf"] Oct 31 22:03:13.065445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.065423 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-855d4sjlf"] Oct 31 22:03:13.784326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.781696 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:13.791325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.787432 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 31 22:03:13.791325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:13.787497 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 31 22:03:14.007565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:14.007522 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00138ef9-1ba8-4578-b64a-72d00ee8bcdd" path="/var/lib/kubelet/pods/00138ef9-1ba8-4578-b64a-72d00ee8bcdd/volumes" Oct 31 22:03:14.783803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:14.783762 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 31 22:03:14.784120 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:14.784087 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 31 22:03:15.788324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:15.781670 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 31 22:03:15.788324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:15.781732 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 31 22:03:16.792328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:16.786670 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 31 22:03:16.792328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:16.786731 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 31 22:03:17.795952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:17.795402 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 31 22:03:17.795952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:17.795468 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 31 22:03:18.780817 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:18.780777 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 31 22:03:18.781052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:18.780859 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 31 22:03:19.781728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:19.781674 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" start-of-body= Oct 31 22:03:19.782162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:19.781768 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": dial tcp 10.134.1.0:15021: connect: connection refused" Oct 31 22:03:20.791671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:20.791636 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:20.792140 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:20.792121 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:20.792889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:20.792873 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:03:22.033571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.033540 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz"] Oct 31 22:03:22.045016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.044988 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ca318a4-ktspz"] Oct 31 22:03:22.977531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.977499 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn"] Oct 31 22:03:22.978136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.978123 4294 cpu_manager.go:414] "containerMap: removing container" podUID="00138ef9-1ba8-4578-b64a-72d00ee8bcdd" containerName="payload-processing" Oct 31 22:03:22.978185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.978137 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00138ef9-1ba8-4578-b64a-72d00ee8bcdd" containerName="payload-processing" Oct 31 22:03:22.978291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.978281 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="00138ef9-1ba8-4578-b64a-72d00ee8bcdd" containerName="payload-processing" Oct 31 22:03:22.983588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.983566 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:22.999799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:22.999774 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn"] Oct 31 22:03:23.006914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.006893 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2cd26837e078\"" Oct 31 22:03:23.092803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.092763 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-routing-overlay-volume\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.093545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.093517 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-plugins-config-volume\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.093734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.093718 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-provider-credentials\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.095592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.095549 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85vn6\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-kube-api-access-85vn6\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.201967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.201929 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-routing-overlay-volume\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.202318 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.202282 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-plugins-config-volume\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.202512 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.202448 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-provider-credentials\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.202669 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.202651 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85vn6\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-kube-api-access-85vn6\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.207327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.203921 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-routing-overlay-volume\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.209572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.208220 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-plugins-config-volume\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.241369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.232461 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85vn6\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-kube-api-access-85vn6\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.241369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.235511 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-provider-credentials\") pod \"payload-processing-external-model-5fb7f7d54b-mfmxn\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.303092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.303063 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:23.814389 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:03:23.814356 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9452397_7366_42a2_bdbe_28088e41080c.slice/crio-8fa8cb8e6378fb9764063d03ffe25d4a5c86d159f5474790d7e4634e8603d830 WatchSource:0}: Error finding container 8fa8cb8e6378fb9764063d03ffe25d4a5c86d159f5474790d7e4634e8603d830: Status 404 returned error can't find the container with id 8fa8cb8e6378fb9764063d03ffe25d4a5c86d159f5474790d7e4634e8603d830 Oct 31 22:03:23.814653 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.814631 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn"] Oct 31 22:03:23.989217 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:23.989185 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1d4569-1867-4943-ba5e-568e8c281339" path="/var/lib/kubelet/pods/8d1d4569-1867-4943-ba5e-568e8c281339/volumes" Oct 31 22:03:24.093724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:24.093627 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" event={"ID":"e9452397-7366-42a2-bdbe-28088e41080c","Type":"ContainerStarted","Data":"d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7"} Oct 31 22:03:24.093724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:24.093693 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:24.093724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:24.093717 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" event={"ID":"e9452397-7366-42a2-bdbe-28088e41080c","Type":"ContainerStarted","Data":"8fa8cb8e6378fb9764063d03ffe25d4a5c86d159f5474790d7e4634e8603d830"} Oct 31 22:03:24.135717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:24.135651 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" podStartSLOduration=2.1356345660000002 podStartE2EDuration="2.135634566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:03:24.135015151 +0000 UTC m=+2844.260122381" watchObservedRunningTime="2026-10-31 22:03:24.135634566 +0000 UTC m=+2844.260741796" Oct 31 22:03:26.897243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.897207 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf"] Oct 31 22:03:26.902081 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.902060 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:26.919097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.919072 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf"] Oct 31 22:03:26.925398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.925372 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:03:26.925565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.925535 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:03:26.966989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.966955 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c89l\" (UniqueName: \"kubernetes.io/projected/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-kube-api-access-7c89l\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:26.967176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.967129 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:26.967262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:26.967248 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.069286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.069248 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.069498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.069381 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.069566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.069546 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c89l\" (UniqueName: \"kubernetes.io/projected/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-kube-api-access-7c89l\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.069945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.069920 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.071743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.071723 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.092629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.092602 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c89l\" (UniqueName: \"kubernetes.io/projected/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-kube-api-access-7c89l\") pod \"maas-api-revoke-sub-8964a30f-w7jcf\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.220513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.220438 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:27.532362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.529908 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf"] Oct 31 22:03:27.532362 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:03:27.530052 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7798a306_9b5b_450c_a3cc_9f9ec3b588ef.slice/crio-e0b8cca8e475428772f8bf67b3ef6e1f98bc28bda1110bdfbbc9cd1088208551 WatchSource:0}: Error finding container e0b8cca8e475428772f8bf67b3ef6e1f98bc28bda1110bdfbbc9cd1088208551: Status 404 returned error can't find the container with id e0b8cca8e475428772f8bf67b3ef6e1f98bc28bda1110bdfbbc9cd1088208551 Oct 31 22:03:27.937216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:27.937184 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn"] Oct 31 22:03:28.194384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:28.174679 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" event={"ID":"7798a306-9b5b-450c-a3cc-9f9ec3b588ef","Type":"ContainerStarted","Data":"a77e2f1e31531978eb868cc25cca388f8a12e7b99b86b217e520da57ed47e6f4"} Oct 31 22:03:28.194384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:28.174748 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" event={"ID":"7798a306-9b5b-450c-a3cc-9f9ec3b588ef","Type":"ContainerStarted","Data":"e0b8cca8e475428772f8bf67b3ef6e1f98bc28bda1110bdfbbc9cd1088208551"} Oct 31 22:03:28.237326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:28.231400 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" podStartSLOduration=2.23137862 podStartE2EDuration="2.23137862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:03:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:03:28.221920025 +0000 UTC m=+2848.347027280" watchObservedRunningTime="2026-10-31 22:03:28.23137862 +0000 UTC m=+2848.356485855" Oct 31 22:03:29.183930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:29.183896 4294 generic.go:356] "Generic (PLEG): container finished" podID="7798a306-9b5b-450c-a3cc-9f9ec3b588ef" containerID="a77e2f1e31531978eb868cc25cca388f8a12e7b99b86b217e520da57ed47e6f4" exitCode=0 Oct 31 22:03:29.184347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:29.183983 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" event={"ID":"7798a306-9b5b-450c-a3cc-9f9ec3b588ef","Type":"ContainerDied","Data":"a77e2f1e31531978eb868cc25cca388f8a12e7b99b86b217e520da57ed47e6f4"} Oct 31 22:03:30.354612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.354586 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:30.436748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.436717 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-openshift-ca\") pod \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " Oct 31 22:03:30.436952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.436936 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-kube-api-access-7c89l\" (UniqueName: \"kubernetes.io/projected/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-kube-api-access-7c89l\") pod \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " Oct 31 22:03:30.437422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.437390 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-openshift-ca" pod "7798a306-9b5b-450c-a3cc-9f9ec3b588ef" (UID: "7798a306-9b5b-450c-a3cc-9f9ec3b588ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:30.437422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.437439 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-xks-ca\") pod \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\" (UID: \"7798a306-9b5b-450c-a3cc-9f9ec3b588ef\") " Oct 31 22:03:30.438157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.437879 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:30.439816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.439784 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-kube-api-access-7c89l" pod "7798a306-9b5b-450c-a3cc-9f9ec3b588ef" (UID: "7798a306-9b5b-450c-a3cc-9f9ec3b588ef"). InnerVolumeSpecName "kube-api-access-7c89l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:30.440002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.439975 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-xks-ca" pod "7798a306-9b5b-450c-a3cc-9f9ec3b588ef" (UID: "7798a306-9b5b-450c-a3cc-9f9ec3b588ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:03:30.539366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.539291 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c89l\" (UniqueName: \"kubernetes.io/projected/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-kube-api-access-7c89l\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:30.539366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:30.539364 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7798a306-9b5b-450c-a3cc-9f9ec3b588ef-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:31.199849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:31.199818 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" Oct 31 22:03:31.200051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:31.199816 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf" event={"ID":"7798a306-9b5b-450c-a3cc-9f9ec3b588ef","Type":"ContainerDied","Data":"e0b8cca8e475428772f8bf67b3ef6e1f98bc28bda1110bdfbbc9cd1088208551"} Oct 31 22:03:31.200051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:31.199966 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0b8cca8e475428772f8bf67b3ef6e1f98bc28bda1110bdfbbc9cd1088208551" Oct 31 22:03:31.311486 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:31.311453 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:03:32.938254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:32.938217 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" podUID="e9452397-7366-42a2-bdbe-28088e41080c" containerName="payload-processing" containerID="cri-o://d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7" gracePeriod=25 Oct 31 22:03:33.045780 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.045745 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd"] Oct 31 22:03:33.049720 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.049696 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ad9afa0-hkcvd"] Oct 31 22:03:33.210159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.210130 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:33.214752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.214727 4294 generic.go:356] "Generic (PLEG): container finished" podID="e9452397-7366-42a2-bdbe-28088e41080c" containerID="d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7" exitCode=0 Oct 31 22:03:33.214901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.214772 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" Oct 31 22:03:33.214901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.214810 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" event={"ID":"e9452397-7366-42a2-bdbe-28088e41080c","Type":"ContainerDied","Data":"d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7"} Oct 31 22:03:33.214901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.214859 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn" event={"ID":"e9452397-7366-42a2-bdbe-28088e41080c","Type":"ContainerDied","Data":"8fa8cb8e6378fb9764063d03ffe25d4a5c86d159f5474790d7e4634e8603d830"} Oct 31 22:03:33.214901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.214881 4294 scope.go:122] "RemoveContainer" containerID="d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7" Oct 31 22:03:33.224737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.224718 4294 scope.go:122] "RemoveContainer" containerID="d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7" Oct 31 22:03:33.224995 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:33.224968 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7\": container with ID starting with d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7 not found: ID does not exist" containerID="d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7" Oct 31 22:03:33.225099 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.224997 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7"} err="failed to get container status \"d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7\": rpc error: code = NotFound desc = could not find container \"d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7\": container with ID starting with d785e72551f34b1c4fe464ac780c39ae33f9380c48574846068b1b16aee243d7 not found: ID does not exist" Oct 31 22:03:33.278193 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.278157 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-plugins-config-volume\") pod \"e9452397-7366-42a2-bdbe-28088e41080c\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " Oct 31 22:03:33.278361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.278218 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-provider-credentials\") pod \"e9452397-7366-42a2-bdbe-28088e41080c\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " Oct 31 22:03:33.278361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.278338 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-routing-overlay-volume\") pod \"e9452397-7366-42a2-bdbe-28088e41080c\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " Oct 31 22:03:33.278498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.278476 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-kube-api-access-85vn6\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-kube-api-access-85vn6\") pod \"e9452397-7366-42a2-bdbe-28088e41080c\" (UID: \"e9452397-7366-42a2-bdbe-28088e41080c\") " Oct 31 22:03:33.278720 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.278680 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-plugins-config-volume" pod "e9452397-7366-42a2-bdbe-28088e41080c" (UID: "e9452397-7366-42a2-bdbe-28088e41080c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:33.278863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.278763 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-routing-overlay-volume" pod "e9452397-7366-42a2-bdbe-28088e41080c" (UID: "e9452397-7366-42a2-bdbe-28088e41080c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:33.279265 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.279240 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:33.279381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.279269 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e9452397-7366-42a2-bdbe-28088e41080c-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:33.281753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.281727 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-provider-credentials" pod "e9452397-7366-42a2-bdbe-28088e41080c" (UID: "e9452397-7366-42a2-bdbe-28088e41080c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:33.283522 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.283494 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-kube-api-access-85vn6" pod "e9452397-7366-42a2-bdbe-28088e41080c" (UID: "e9452397-7366-42a2-bdbe-28088e41080c"). InnerVolumeSpecName "kube-api-access-85vn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:33.380765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.380727 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:33.380765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.380759 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85vn6\" (UniqueName: \"kubernetes.io/projected/e9452397-7366-42a2-bdbe-28088e41080c-kube-api-access-85vn6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:33.563458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.563425 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn"] Oct 31 22:03:33.574816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:33.574786 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-mfmxn"] Oct 31 22:03:34.020750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:34.020715 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37fd1f9-fa35-48f3-82c8-078ecd00d47f" path="/var/lib/kubelet/pods/c37fd1f9-fa35-48f3-82c8-078ecd00d47f/volumes" Oct 31 22:03:34.035235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:34.035201 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9452397-7366-42a2-bdbe-28088e41080c" path="/var/lib/kubelet/pods/e9452397-7366-42a2-bdbe-28088e41080c/volumes" Oct 31 22:03:35.248664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.248626 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7"] Oct 31 22:03:35.249343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.249324 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e9452397-7366-42a2-bdbe-28088e41080c" containerName="payload-processing" Oct 31 22:03:35.249389 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.249346 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9452397-7366-42a2-bdbe-28088e41080c" containerName="payload-processing" Oct 31 22:03:35.249389 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.249364 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7798a306-9b5b-450c-a3cc-9f9ec3b588ef" containerName="revoke-keys" Oct 31 22:03:35.249389 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.249371 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7798a306-9b5b-450c-a3cc-9f9ec3b588ef" containerName="revoke-keys" Oct 31 22:03:35.249536 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.249522 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9452397-7366-42a2-bdbe-28088e41080c" containerName="payload-processing" Oct 31 22:03:35.249573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.249542 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7798a306-9b5b-450c-a3cc-9f9ec3b588ef" containerName="revoke-keys" Oct 31 22:03:35.254639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.254619 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.273615 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.273587 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7"] Oct 31 22:03:35.314404 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.314366 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2cd26837e078\"" Oct 31 22:03:35.315521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.315488 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-provider-credentials\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.315674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.315551 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhx29\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-kube-api-access-fhx29\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.315717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.315691 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-plugins-config-volume\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.315807 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.315789 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-routing-overlay-volume\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.417018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.416978 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-provider-credentials\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.417194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.417031 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhx29\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-kube-api-access-fhx29\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.417194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.417080 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-plugins-config-volume\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.417194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.417142 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-routing-overlay-volume\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.417837 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.417809 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-routing-overlay-volume\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.418047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.418022 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-plugins-config-volume\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.419638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.419614 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-provider-credentials\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.441838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.441816 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhx29\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-kube-api-access-fhx29\") pod \"payload-processing-external-model-5fb7f7d54b-fkfs7\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.569941 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.569862 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:35.873882 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:03:35.873849 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401c30c0_c71b_4cf2_a5b2_4f73e09155fe.slice/crio-32c96f7211a979a087c1943fb9e0e11a25f2ff013e8a58d564872c009b466f04 WatchSource:0}: Error finding container 32c96f7211a979a087c1943fb9e0e11a25f2ff013e8a58d564872c009b466f04: Status 404 returned error can't find the container with id 32c96f7211a979a087c1943fb9e0e11a25f2ff013e8a58d564872c009b466f04 Oct 31 22:03:35.873882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:35.873857 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7"] Oct 31 22:03:36.336315 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:36.336276 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" event={"ID":"401c30c0-c71b-4cf2-a5b2-4f73e09155fe","Type":"ContainerStarted","Data":"b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb"} Oct 31 22:03:36.336739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:36.336354 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" event={"ID":"401c30c0-c71b-4cf2-a5b2-4f73e09155fe","Type":"ContainerStarted","Data":"32c96f7211a979a087c1943fb9e0e11a25f2ff013e8a58d564872c009b466f04"} Oct 31 22:03:36.336739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:36.336376 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:36.380327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:36.380271 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" podStartSLOduration=1.380258039 podStartE2EDuration="1.380258039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:03:36.37761671 +0000 UTC m=+2856.502723949" watchObservedRunningTime="2026-10-31 22:03:36.380258039 +0000 UTC m=+2856.505365269" Oct 31 22:03:37.582714 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.582678 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j"] Oct 31 22:03:37.587045 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.587025 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.604390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.604361 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j"] Oct 31 22:03:37.611312 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.611278 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:03:37.611462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.611432 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:03:37.647043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.647008 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.647197 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.647144 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.647273 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.647209 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggrh\" (UniqueName: \"kubernetes.io/projected/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-kube-api-access-hggrh\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.749192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.749154 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.749396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.749234 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggrh\" (UniqueName: \"kubernetes.io/projected/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-kube-api-access-hggrh\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.749468 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.749451 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.750083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.750056 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.751546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.751524 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.772741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.772708 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggrh\" (UniqueName: \"kubernetes.io/projected/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-kube-api-access-hggrh\") pod \"maas-api-revoke-sub-77fecb57-r5x4j\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:37.902201 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:37.902122 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:38.231288 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:03:38.231252 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5f0e45b_f1ca_407d_a3fa_d6e3af734bfe.slice/crio-a46e0e4348050f09ae44339458502c485a06f9b2bae4700b7ea7e0d1cd0aa799 WatchSource:0}: Error finding container a46e0e4348050f09ae44339458502c485a06f9b2bae4700b7ea7e0d1cd0aa799: Status 404 returned error can't find the container with id a46e0e4348050f09ae44339458502c485a06f9b2bae4700b7ea7e0d1cd0aa799 Oct 31 22:03:38.231455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:38.231432 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j"] Oct 31 22:03:38.351791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:38.351759 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" event={"ID":"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe","Type":"ContainerStarted","Data":"a46e0e4348050f09ae44339458502c485a06f9b2bae4700b7ea7e0d1cd0aa799"} Oct 31 22:03:39.360399 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:39.360292 4294 generic.go:356] "Generic (PLEG): container finished" podID="e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" containerID="e15fac2aed376b85b3c12ce2bfb1cee7fe7efaec51c41730920a0abf94c593ee" exitCode=0 Oct 31 22:03:39.360399 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:39.360349 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" event={"ID":"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe","Type":"ContainerDied","Data":"e15fac2aed376b85b3c12ce2bfb1cee7fe7efaec51c41730920a0abf94c593ee"} Oct 31 22:03:40.496462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.496430 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7"] Oct 31 22:03:40.499196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.499176 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:40.527260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.527230 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:40.592963 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.592930 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-xks-ca\") pod \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " Oct 31 22:03:40.593157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.593002 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-openshift-ca\") pod \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " Oct 31 22:03:40.593257 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.593237 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-kube-api-access-hggrh\" (UniqueName: \"kubernetes.io/projected/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-kube-api-access-hggrh\") pod \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\" (UID: \"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe\") " Oct 31 22:03:40.593718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.593682 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-openshift-ca" pod "e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" (UID: "e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:40.593885 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.593860 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:40.595610 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.595584 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-kube-api-access-hggrh" pod "e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" (UID: "e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe"). InnerVolumeSpecName "kube-api-access-hggrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:40.598081 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.598054 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-xks-ca" pod "e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" (UID: "e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:03:40.694985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.694904 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hggrh\" (UniqueName: \"kubernetes.io/projected/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-kube-api-access-hggrh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:40.694985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:40.694936 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:41.376801 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:41.376774 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" Oct 31 22:03:41.376998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:41.376771 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j" event={"ID":"e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe","Type":"ContainerDied","Data":"a46e0e4348050f09ae44339458502c485a06f9b2bae4700b7ea7e0d1cd0aa799"} Oct 31 22:03:41.376998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:41.376899 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a46e0e4348050f09ae44339458502c485a06f9b2bae4700b7ea7e0d1cd0aa799" Oct 31 22:03:43.365093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.365054 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb"] Oct 31 22:03:43.365736 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.365722 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" containerName="revoke-keys" Oct 31 22:03:43.365794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.365737 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" containerName="revoke-keys" Oct 31 22:03:43.365846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.365835 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" containerName="revoke-keys" Oct 31 22:03:43.369591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.369574 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.387999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.387969 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb"] Oct 31 22:03:43.393822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.393797 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-253d07fe4854\"" Oct 31 22:03:43.451147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.451113 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-provider-credentials\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.451343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.451260 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-plugins-config-volume\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.451408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.451400 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-routing-overlay-volume\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.451452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.451425 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snm95\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-kube-api-access-snm95\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.552732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.552691 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-plugins-config-volume\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.552922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.552819 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-routing-overlay-volume\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.552922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.552849 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snm95\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-kube-api-access-snm95\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.552922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.552910 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-provider-credentials\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.553422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.553395 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-plugins-config-volume\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.553523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.553466 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-routing-overlay-volume\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.555408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.555380 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-provider-credentials\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.576857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.576821 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snm95\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-kube-api-access-snm95\") pod \"payload-processing-external-model-86bc8645b9-btzdb\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.684015 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.683933 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:43.990473 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:03:43.990449 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8ac4f2_e3b3_462a_99ea_674d9a598e3c.slice/crio-847973f4d824497e94bb7be1337ca8122f921e0a459af9221007e5d56eeede17 WatchSource:0}: Error finding container 847973f4d824497e94bb7be1337ca8122f921e0a459af9221007e5d56eeede17: Status 404 returned error can't find the container with id 847973f4d824497e94bb7be1337ca8122f921e0a459af9221007e5d56eeede17 Oct 31 22:03:43.991264 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:43.991230 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb"] Oct 31 22:03:44.438540 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:44.438449 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" event={"ID":"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c","Type":"ContainerStarted","Data":"5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c"} Oct 31 22:03:44.438540 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:44.438499 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" event={"ID":"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c","Type":"ContainerStarted","Data":"847973f4d824497e94bb7be1337ca8122f921e0a459af9221007e5d56eeede17"} Oct 31 22:03:44.438540 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:44.438536 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:44.482802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:44.482755 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" podStartSLOduration=1.482741641 podStartE2EDuration="1.482741641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:03:44.479824291 +0000 UTC m=+2864.604931522" watchObservedRunningTime="2026-10-31 22:03:44.482741641 +0000 UTC m=+2864.607848870" Oct 31 22:03:45.034597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.034567 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf"] Oct 31 22:03:45.046778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.046751 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d2e706d-bz8pf"] Oct 31 22:03:45.497263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.497208 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" podUID="401c30c0-c71b-4cf2-a5b2-4f73e09155fe" containerName="payload-processing" containerID="cri-o://b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb" gracePeriod=25 Oct 31 22:03:45.773236 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.773210 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:45.885394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.885361 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-routing-overlay-volume\") pod \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " Oct 31 22:03:45.885619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.885432 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-plugins-config-volume\") pod \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " Oct 31 22:03:45.885619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.885486 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-provider-credentials\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-provider-credentials\") pod \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " Oct 31 22:03:45.885814 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.885790 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-routing-overlay-volume" pod "401c30c0-c71b-4cf2-a5b2-4f73e09155fe" (UID: "401c30c0-c71b-4cf2-a5b2-4f73e09155fe"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:45.885860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.885841 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-kube-api-access-fhx29\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-kube-api-access-fhx29\") pod \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\" (UID: \"401c30c0-c71b-4cf2-a5b2-4f73e09155fe\") " Oct 31 22:03:45.885860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.885837 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-plugins-config-volume" pod "401c30c0-c71b-4cf2-a5b2-4f73e09155fe" (UID: "401c30c0-c71b-4cf2-a5b2-4f73e09155fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:45.886355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.886336 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:45.886355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.886357 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:45.887700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.887679 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-provider-credentials" pod "401c30c0-c71b-4cf2-a5b2-4f73e09155fe" (UID: "401c30c0-c71b-4cf2-a5b2-4f73e09155fe"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:45.887881 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.887860 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-kube-api-access-fhx29" pod "401c30c0-c71b-4cf2-a5b2-4f73e09155fe" (UID: "401c30c0-c71b-4cf2-a5b2-4f73e09155fe"). InnerVolumeSpecName "kube-api-access-fhx29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:45.988660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.988627 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhx29\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-kube-api-access-fhx29\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:45.988660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:45.988657 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/401c30c0-c71b-4cf2-a5b2-4f73e09155fe-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:46.004629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.004551 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827a8b47-7adf-4b07-be7c-fb4dc2bdb767" path="/var/lib/kubelet/pods/827a8b47-7adf-4b07-be7c-fb4dc2bdb767/volumes" Oct 31 22:03:46.455399 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.455294 4294 generic.go:356] "Generic (PLEG): container finished" podID="401c30c0-c71b-4cf2-a5b2-4f73e09155fe" containerID="b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb" exitCode=0 Oct 31 22:03:46.455399 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.455366 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" event={"ID":"401c30c0-c71b-4cf2-a5b2-4f73e09155fe","Type":"ContainerDied","Data":"b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb"} Oct 31 22:03:46.455601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.455411 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" Oct 31 22:03:46.455601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.455448 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7" event={"ID":"401c30c0-c71b-4cf2-a5b2-4f73e09155fe","Type":"ContainerDied","Data":"32c96f7211a979a087c1943fb9e0e11a25f2ff013e8a58d564872c009b466f04"} Oct 31 22:03:46.455601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.455471 4294 scope.go:122] "RemoveContainer" containerID="b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb" Oct 31 22:03:46.467555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.467533 4294 scope.go:122] "RemoveContainer" containerID="b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb" Oct 31 22:03:46.467863 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:46.467840 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb\": container with ID starting with b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb not found: ID does not exist" containerID="b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb" Oct 31 22:03:46.467941 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.467873 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb"} err="failed to get container status \"b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb\": rpc error: code = NotFound desc = could not find container \"b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb\": container with ID starting with b4604109d988e58653f7d5a606361ef1c85493841d00464873de70542a6b2edb not found: ID does not exist" Oct 31 22:03:46.509308 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.509264 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7"] Oct 31 22:03:46.520984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:46.520956 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5fb7f7d54b-fkfs7"] Oct 31 22:03:47.986095 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:47.986059 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401c30c0-c71b-4cf2-a5b2-4f73e09155fe" path="/var/lib/kubelet/pods/401c30c0-c71b-4cf2-a5b2-4f73e09155fe/volumes" Oct 31 22:03:48.627226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:48.627188 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb"] Oct 31 22:03:50.408753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.408716 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd"] Oct 31 22:03:50.409453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.409438 4294 cpu_manager.go:414] "containerMap: removing container" podUID="401c30c0-c71b-4cf2-a5b2-4f73e09155fe" containerName="payload-processing" Oct 31 22:03:50.409504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.409454 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401c30c0-c71b-4cf2-a5b2-4f73e09155fe" containerName="payload-processing" Oct 31 22:03:50.409572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.409562 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="401c30c0-c71b-4cf2-a5b2-4f73e09155fe" containerName="payload-processing" Oct 31 22:03:50.414546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.414524 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.435344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.430535 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd"] Oct 31 22:03:50.544326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.527190 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.544739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.544705 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.555331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.544964 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.555754 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.555724 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24hnm\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-kube-api-access-24hnm\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.657192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.657153 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.657507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.657483 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.657621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.657558 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24hnm\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-kube-api-access-24hnm\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.657792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.657762 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.657980 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:50.657963 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696f9a/agc-cred-ed5d4be8dd27: secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:50.658075 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:50.657982 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd: secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:50.658075 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:50.658046 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials podName:0c79e0c9-8a84-4576-b382-1c78dc0d71af nodeName:}" failed. No retries permitted until 2026-10-31 22:03:51.158025705 +0000 UTC m=+2871.283132930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" (UID: "0c79e0c9-8a84-4576-b382-1c78dc0d71af") : secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:50.658606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.658508 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.659425 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.659374 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:50.681414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:50.681390 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24hnm\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-kube-api-access-24hnm\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:51.168213 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:51.168158 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:51.168987 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:51.168961 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696f9a/agc-cred-ed5d4be8dd27: secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:51.169128 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:51.169117 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd: secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:51.169321 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:51.169295 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials podName:0c79e0c9-8a84-4576-b382-1c78dc0d71af nodeName:}" failed. No retries permitted until 2026-10-31 22:03:52.169273818 +0000 UTC m=+2872.294381046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" (UID: "0c79e0c9-8a84-4576-b382-1c78dc0d71af") : secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:51.894640 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:51.894605 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd"] Oct 31 22:03:51.895193 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:51.895018 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" podUID="0c79e0c9-8a84-4576-b382-1c78dc0d71af" Oct 31 22:03:51.908822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:51.908717 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn"] Oct 31 22:03:52.187173 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.187075 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:52.187361 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:52.187253 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-696f9a/agc-cred-ed5d4be8dd27: secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:52.187361 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:52.187276 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd: secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:52.187442 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:52.187395 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials podName:0c79e0c9-8a84-4576-b382-1c78dc0d71af nodeName:}" failed. No retries permitted until 2026-10-31 22:03:54.187373459 +0000 UTC m=+2874.312480684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" (UID: "0c79e0c9-8a84-4576-b382-1c78dc0d71af") : secret "agc-cred-ed5d4be8dd27" not found Oct 31 22:03:52.577040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.577006 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:52.585477 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.585452 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:52.707766 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.707736 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-routing-overlay-volume\") pod \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " Oct 31 22:03:52.707962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.707785 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-plugins-config-volume\") pod \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " Oct 31 22:03:52.708077 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.708056 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-kube-api-access-24hnm\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-kube-api-access-24hnm\") pod \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\" (UID: \"0c79e0c9-8a84-4576-b382-1c78dc0d71af\") " Oct 31 22:03:52.708171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.708141 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-routing-overlay-volume" pod "0c79e0c9-8a84-4576-b382-1c78dc0d71af" (UID: "0c79e0c9-8a84-4576-b382-1c78dc0d71af"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:52.708271 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.708251 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-plugins-config-volume" pod "0c79e0c9-8a84-4576-b382-1c78dc0d71af" (UID: "0c79e0c9-8a84-4576-b382-1c78dc0d71af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:52.708672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.708654 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:52.708717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.708677 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c79e0c9-8a84-4576-b382-1c78dc0d71af-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:52.710203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.710185 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-kube-api-access-24hnm" pod "0c79e0c9-8a84-4576-b382-1c78dc0d71af" (UID: "0c79e0c9-8a84-4576-b382-1c78dc0d71af"). InnerVolumeSpecName "kube-api-access-24hnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:52.810289 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:52.810252 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24hnm\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-kube-api-access-24hnm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:53.584359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:53.584329 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd" Oct 31 22:03:53.656425 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:53.648881 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" podUID="0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" containerName="payload-processing" containerID="cri-o://5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c" gracePeriod=25 Oct 31 22:03:53.793391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:53.793349 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd"] Oct 31 22:03:53.799435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:53.799404 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5cd9968jd"] Oct 31 22:03:53.955175 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:53.955136 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0c79e0c9-8a84-4576-b382-1c78dc0d71af-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:53.987043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:53.987008 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c79e0c9-8a84-4576-b382-1c78dc0d71af" path="/var/lib/kubelet/pods/0c79e0c9-8a84-4576-b382-1c78dc0d71af/volumes" Oct 31 22:03:54.047208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.047184 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:54.157852 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.157767 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-kube-api-access-snm95\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-kube-api-access-snm95\") pod \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " Oct 31 22:03:54.158022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.157860 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-provider-credentials\") pod \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " Oct 31 22:03:54.158022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.157952 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-routing-overlay-volume\") pod \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " Oct 31 22:03:54.158105 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.158089 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-plugins-config-volume\") pod \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\" (UID: \"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c\") " Oct 31 22:03:54.158325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.158283 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-routing-overlay-volume" pod "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" (UID: "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:54.158553 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.158528 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-plugins-config-volume" pod "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" (UID: "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:54.158746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.158731 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:54.158795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.158749 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:54.159999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.159979 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-kube-api-access-snm95" pod "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" (UID: "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c"). InnerVolumeSpecName "kube-api-access-snm95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:54.160073 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.160008 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-provider-credentials" pod "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" (UID: "0a8ac4f2-e3b3-462a-99ea-674d9a598e3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:54.259727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.259687 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snm95\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-kube-api-access-snm95\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:54.259727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.259723 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:54.593318 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.593270 4294 generic.go:356] "Generic (PLEG): container finished" podID="0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" containerID="5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c" exitCode=0 Oct 31 22:03:54.593864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.593327 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" event={"ID":"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c","Type":"ContainerDied","Data":"5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c"} Oct 31 22:03:54.593864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.593340 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" Oct 31 22:03:54.593864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.593370 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb" event={"ID":"0a8ac4f2-e3b3-462a-99ea-674d9a598e3c","Type":"ContainerDied","Data":"847973f4d824497e94bb7be1337ca8122f921e0a459af9221007e5d56eeede17"} Oct 31 22:03:54.593864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.593394 4294 scope.go:122] "RemoveContainer" containerID="5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c" Oct 31 22:03:54.604666 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.604647 4294 scope.go:122] "RemoveContainer" containerID="5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c" Oct 31 22:03:54.604935 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:54.604916 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c\": container with ID starting with 5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c not found: ID does not exist" containerID="5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c" Oct 31 22:03:54.605010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.604941 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c"} err="failed to get container status \"5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c\": rpc error: code = NotFound desc = could not find container \"5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c\": container with ID starting with 5c4991ed07879d8f14fde5b5199472045ba8f6e7a7451e7b23f547e38adcbd1c not found: ID does not exist" Oct 31 22:03:54.650138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.650103 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb"] Oct 31 22:03:54.661914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:54.661881 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-86bc8645b9-btzdb"] Oct 31 22:03:55.986464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:55.986430 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" path="/var/lib/kubelet/pods/0a8ac4f2-e3b3-462a-99ea-674d9a598e3c/volumes" Oct 31 22:03:56.913324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:56.911442 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" podUID="0df2ebf3-7b3f-49f9-97da-a889523e2b51" containerName="payload-processing" containerID="cri-o://f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce" gracePeriod=25 Oct 31 22:03:57.267966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.267599 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:03:57.344322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.343932 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-kube-api-access-hnnzf\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-kube-api-access-hnnzf\") pod \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " Oct 31 22:03:57.344322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.344003 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-provider-credentials\") pod \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " Oct 31 22:03:57.344322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.344063 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-routing-overlay-volume\") pod \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " Oct 31 22:03:57.344322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.344121 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-plugins-config-volume\") pod \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\" (UID: \"0df2ebf3-7b3f-49f9-97da-a889523e2b51\") " Oct 31 22:03:57.358206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.348903 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-plugins-config-volume" pod "0df2ebf3-7b3f-49f9-97da-a889523e2b51" (UID: "0df2ebf3-7b3f-49f9-97da-a889523e2b51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:57.358206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.349244 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-routing-overlay-volume" pod "0df2ebf3-7b3f-49f9-97da-a889523e2b51" (UID: "0df2ebf3-7b3f-49f9-97da-a889523e2b51"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:03:57.358206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.352042 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:57.358206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.352068 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0df2ebf3-7b3f-49f9-97da-a889523e2b51-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:57.366922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.366869 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-provider-credentials" pod "0df2ebf3-7b3f-49f9-97da-a889523e2b51" (UID: "0df2ebf3-7b3f-49f9-97da-a889523e2b51"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:57.461381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.446490 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-kube-api-access-hnnzf" pod "0df2ebf3-7b3f-49f9-97da-a889523e2b51" (UID: "0df2ebf3-7b3f-49f9-97da-a889523e2b51"). InnerVolumeSpecName "kube-api-access-hnnzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:03:57.461381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.453628 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnnzf\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-kube-api-access-hnnzf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:57.461381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.453662 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0df2ebf3-7b3f-49f9-97da-a889523e2b51-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:03:57.625974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.625932 4294 generic.go:356] "Generic (PLEG): container finished" podID="0df2ebf3-7b3f-49f9-97da-a889523e2b51" containerID="f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce" exitCode=0 Oct 31 22:03:57.626183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.626161 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" event={"ID":"0df2ebf3-7b3f-49f9-97da-a889523e2b51","Type":"ContainerDied","Data":"f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce"} Oct 31 22:03:57.626237 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.626223 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" event={"ID":"0df2ebf3-7b3f-49f9-97da-a889523e2b51","Type":"ContainerDied","Data":"e6a36ece321fda71c36e3f5b9aebde549d74fedcad42226e039826ad50208a16"} Oct 31 22:03:57.626280 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.626258 4294 scope.go:122] "RemoveContainer" containerID="f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce" Oct 31 22:03:57.626483 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.626466 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn" Oct 31 22:03:57.642746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.642723 4294 scope.go:122] "RemoveContainer" containerID="f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce" Oct 31 22:03:57.643139 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:03:57.643115 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce\": container with ID starting with f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce not found: ID does not exist" containerID="f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce" Oct 31 22:03:57.643244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.643150 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce"} err="failed to get container status \"f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce\": rpc error: code = NotFound desc = could not find container \"f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce\": container with ID starting with f0e765216928502e9b57b9edcec320e3a33c99d707637887ae85a79284c000ce not found: ID does not exist" Oct 31 22:03:57.686704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.686672 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn"] Oct 31 22:03:57.690438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.690413 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-5455787hn"] Oct 31 22:03:57.989023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:57.988973 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0df2ebf3-7b3f-49f9-97da-a889523e2b51" path="/var/lib/kubelet/pods/0df2ebf3-7b3f-49f9-97da-a889523e2b51/volumes" Oct 31 22:03:59.565576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:59.565442 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-c7qkl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:03:59.565576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:03:59.565488 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-c7qkl" podUID="ddb69f48-47a6-492f-9564-33f725285f09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:04:01.310759 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:04:01.310721 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:04:02.647148 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:02.647111 4294 scope.go:122] "RemoveContainer" containerID="0f156f9b54765aa01996cd9ec6187eac76a08941613084921224ead9997c577c" Oct 31 22:04:07.043000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.042958 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5"] Oct 31 22:04:07.055122 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.055084 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73b4f1eb-mbpf5"] Oct 31 22:04:07.578819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.578786 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc"] Oct 31 22:04:07.579433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.579419 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0df2ebf3-7b3f-49f9-97da-a889523e2b51" containerName="payload-processing" Oct 31 22:04:07.579493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.579434 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0df2ebf3-7b3f-49f9-97da-a889523e2b51" containerName="payload-processing" Oct 31 22:04:07.579493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.579480 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" containerName="payload-processing" Oct 31 22:04:07.579493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.579485 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" containerName="payload-processing" Oct 31 22:04:07.579605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.579583 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8ac4f2-e3b3-462a-99ea-674d9a598e3c" containerName="payload-processing" Oct 31 22:04:07.579605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.579594 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0df2ebf3-7b3f-49f9-97da-a889523e2b51" containerName="payload-processing" Oct 31 22:04:07.584686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.584670 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.618391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.618363 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"agc-cred-c566851d8db1\"" Oct 31 22:04:07.618550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.618535 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"payload-processing-external-model-e2e--23c252ba-dockercfg-7tkpq\"" Oct 31 22:04:07.626422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.626395 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc"] Oct 31 22:04:07.630424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.630402 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"openshift-service-ca.crt\"" Oct 31 22:04:07.630569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.630481 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"kube-root-ca.crt\"" Oct 31 22:04:07.630569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.630495 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"routing-overlay\"" Oct 31 22:04:07.631955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.631932 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4a73b1\"" Oct 31 22:04:07.711656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.711623 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.711836 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.711793 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.711893 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.711841 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmxfp\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-kube-api-access-jmxfp\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.712124 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.712106 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.826716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.826669 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.828961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.828892 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmxfp\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-kube-api-access-jmxfp\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.829409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.829384 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.829663 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.829642 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.855134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.836129 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.855134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.839024 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.857007 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.856976 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.892089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.892049 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmxfp\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-kube-api-access-jmxfp\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.899678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.899645 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:07.992025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:07.990657 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42d31e42-bc58-4c0f-b59a-3f48eb4608ad" path="/var/lib/kubelet/pods/42d31e42-bc58-4c0f-b59a-3f48eb4608ad/volumes" Oct 31 22:04:08.226594 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.226560 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc"] Oct 31 22:04:08.229396 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:04:08.227981 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59404ca5_f2a4_48a4_9098_5b0c65814957.slice/crio-6e690b368b8cfae669bc4a9478febd99f087813e30ad119e833eb054c444e0d2 WatchSource:0}: Error finding container 6e690b368b8cfae669bc4a9478febd99f087813e30ad119e833eb054c444e0d2: Status 404 returned error can't find the container with id 6e690b368b8cfae669bc4a9478febd99f087813e30ad119e833eb054c444e0d2 Oct 31 22:04:08.584210 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.584174 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq"] Oct 31 22:04:08.588603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.588582 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.606176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.606151 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq"] Oct 31 22:04:08.616346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.613882 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"openshift-service-ca.crt\"" Oct 31 22:04:08.616346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.613972 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"routing-overlay\"" Oct 31 22:04:08.616346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.613973 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-e2e--956cda1a-dockercfg-mccg7\"" Oct 31 22:04:08.616346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.614042 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"agc-cred-ca74aeda362a\"" Oct 31 22:04:08.616346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.614085 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-696f9a\"" Oct 31 22:04:08.616346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.615373 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"kube-root-ca.crt\"" Oct 31 22:04:08.659157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.659127 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.659157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.659159 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.659375 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.659292 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.659518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.659500 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tml7\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-kube-api-access-4tml7\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.765437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.765384 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tml7\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-kube-api-access-4tml7\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.765910 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.765875 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.766082 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.766064 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.766257 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.766240 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.776326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.767402 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.777237 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.777038 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.792181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.792149 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.852372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.838996 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tml7\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-kube-api-access-4tml7\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:08.855324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.853877 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" event={"ID":"59404ca5-f2a4-48a4-9098-5b0c65814957","Type":"ContainerStarted","Data":"49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc"} Oct 31 22:04:08.855324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.853956 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" event={"ID":"59404ca5-f2a4-48a4-9098-5b0c65814957","Type":"ContainerStarted","Data":"6e690b368b8cfae669bc4a9478febd99f087813e30ad119e833eb054c444e0d2"} Oct 31 22:04:08.855324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.855247 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:08.899735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.899671 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" podStartSLOduration=1.899651539 podStartE2EDuration="1.899651539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:04:08.896228206 +0000 UTC m=+2889.021335450" watchObservedRunningTime="2026-10-31 22:04:08.899651539 +0000 UTC m=+2889.024758771" Oct 31 22:04:08.903609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:08.903575 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:09.214718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:09.214692 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq"] Oct 31 22:04:09.215239 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:04:09.215211 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28024b1e_f5cd_48a3_b7ce_53b6f2403c29.slice/crio-bfd0c650e8e8940434d4105a12bd812bba49e4b843f3fe9e19ee6b3b5d4cff2f WatchSource:0}: Error finding container bfd0c650e8e8940434d4105a12bd812bba49e4b843f3fe9e19ee6b3b5d4cff2f: Status 404 returned error can't find the container with id bfd0c650e8e8940434d4105a12bd812bba49e4b843f3fe9e19ee6b3b5d4cff2f Oct 31 22:04:09.863777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:09.863737 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" event={"ID":"28024b1e-f5cd-48a3-b7ce-53b6f2403c29","Type":"ContainerStarted","Data":"c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085"} Oct 31 22:04:09.864200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:09.863801 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:09.864200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:09.863817 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" event={"ID":"28024b1e-f5cd-48a3-b7ce-53b6f2403c29","Type":"ContainerStarted","Data":"bfd0c650e8e8940434d4105a12bd812bba49e4b843f3fe9e19ee6b3b5d4cff2f"} Oct 31 22:04:09.909658 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:09.909609 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" podStartSLOduration=1.909595569 podStartE2EDuration="1.909595569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:04:09.90549619 +0000 UTC m=+2890.030603431" watchObservedRunningTime="2026-10-31 22:04:09.909595569 +0000 UTC m=+2890.034702800" Oct 31 22:04:11.831604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.831568 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7"] Oct 31 22:04:11.836217 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.836198 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:11.853117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.853093 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7"] Oct 31 22:04:11.861339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.860572 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:04:11.861339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.860606 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:04:11.923242 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.923213 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:11.923438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.923286 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:11.923496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:11.923470 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgzt\" (UniqueName: \"kubernetes.io/projected/fb51e128-721e-43ec-a9e5-367d78845641-kube-api-access-dmgzt\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.025321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.025279 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.025520 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.025391 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.025597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.025580 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgzt\" (UniqueName: \"kubernetes.io/projected/fb51e128-721e-43ec-a9e5-367d78845641-kube-api-access-dmgzt\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.026050 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.026019 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.027735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.027715 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.048958 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.048931 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgzt\" (UniqueName: \"kubernetes.io/projected/fb51e128-721e-43ec-a9e5-367d78845641-kube-api-access-dmgzt\") pod \"maas-api-revoke-sub-073d4981-pfzf7\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.150720 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.150638 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:12.454494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.454463 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7"] Oct 31 22:04:12.455098 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:04:12.455068 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb51e128_721e_43ec_a9e5_367d78845641.slice/crio-7b53db248e73177e251d10337cfcb349e11a55fce7f6dc7a8f88fa8e56c25b57 WatchSource:0}: Error finding container 7b53db248e73177e251d10337cfcb349e11a55fce7f6dc7a8f88fa8e56c25b57: Status 404 returned error can't find the container with id 7b53db248e73177e251d10337cfcb349e11a55fce7f6dc7a8f88fa8e56c25b57 Oct 31 22:04:12.891154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.891113 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" event={"ID":"fb51e128-721e-43ec-a9e5-367d78845641","Type":"ContainerStarted","Data":"60733d7bee8c0397bc48b9859b852bc756da4213cb2d913f879b697a11ea1046"} Oct 31 22:04:12.891642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.891176 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" event={"ID":"fb51e128-721e-43ec-a9e5-367d78845641","Type":"ContainerStarted","Data":"7b53db248e73177e251d10337cfcb349e11a55fce7f6dc7a8f88fa8e56c25b57"} Oct 31 22:04:12.932583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:12.932536 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" podStartSLOduration=1.9325213959999998 podStartE2EDuration="1.932521396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:04:12.931111893 +0000 UTC m=+2893.056219125" watchObservedRunningTime="2026-10-31 22:04:12.932521396 +0000 UTC m=+2893.057628626" Oct 31 22:04:13.899719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:13.899683 4294 generic.go:356] "Generic (PLEG): container finished" podID="fb51e128-721e-43ec-a9e5-367d78845641" containerID="60733d7bee8c0397bc48b9859b852bc756da4213cb2d913f879b697a11ea1046" exitCode=0 Oct 31 22:04:13.900201 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:13.899760 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" event={"ID":"fb51e128-721e-43ec-a9e5-367d78845641","Type":"ContainerDied","Data":"60733d7bee8c0397bc48b9859b852bc756da4213cb2d913f879b697a11ea1046"} Oct 31 22:04:15.067509 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.067481 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:15.169142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.169045 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-xks-ca\") pod \"fb51e128-721e-43ec-a9e5-367d78845641\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " Oct 31 22:04:15.169361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.169148 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb51e128-721e-43ec-a9e5-367d78845641-kube-api-access-dmgzt\" (UniqueName: \"kubernetes.io/projected/fb51e128-721e-43ec-a9e5-367d78845641-kube-api-access-dmgzt\") pod \"fb51e128-721e-43ec-a9e5-367d78845641\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " Oct 31 22:04:15.169463 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.169360 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-openshift-ca\") pod \"fb51e128-721e-43ec-a9e5-367d78845641\" (UID: \"fb51e128-721e-43ec-a9e5-367d78845641\") " Oct 31 22:04:15.169791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.169757 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-openshift-ca" pod "fb51e128-721e-43ec-a9e5-367d78845641" (UID: "fb51e128-721e-43ec-a9e5-367d78845641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:04:15.170099 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.170074 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:15.171485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.171462 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-xks-ca" pod "fb51e128-721e-43ec-a9e5-367d78845641" (UID: "fb51e128-721e-43ec-a9e5-367d78845641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:04:15.171579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.171501 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb51e128-721e-43ec-a9e5-367d78845641-kube-api-access-dmgzt" pod "fb51e128-721e-43ec-a9e5-367d78845641" (UID: "fb51e128-721e-43ec-a9e5-367d78845641"). InnerVolumeSpecName "kube-api-access-dmgzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:04:15.270874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.270825 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb51e128-721e-43ec-a9e5-367d78845641-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:15.270874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.270870 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmgzt\" (UniqueName: \"kubernetes.io/projected/fb51e128-721e-43ec-a9e5-367d78845641-kube-api-access-dmgzt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:15.916841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.916811 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" Oct 31 22:04:15.917042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.916835 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7" event={"ID":"fb51e128-721e-43ec-a9e5-367d78845641","Type":"ContainerDied","Data":"7b53db248e73177e251d10337cfcb349e11a55fce7f6dc7a8f88fa8e56c25b57"} Oct 31 22:04:15.917042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:15.916907 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b53db248e73177e251d10337cfcb349e11a55fce7f6dc7a8f88fa8e56c25b57" Oct 31 22:04:19.033803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:19.033761 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m"] Oct 31 22:04:19.046564 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:19.046532 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ace7a410-h9j4m"] Oct 31 22:04:19.995150 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:19.995111 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09868d24-c0a6-4979-af86-1b256e50eb3f" path="/var/lib/kubelet/pods/09868d24-c0a6-4979-af86-1b256e50eb3f/volumes" Oct 31 22:04:20.875597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:20.875566 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:04:20.876003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:20.875643 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:04:23.164293 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:23.164259 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs"] Oct 31 22:04:26.364186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.364146 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq"] Oct 31 22:04:26.364974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.364951 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fb51e128-721e-43ec-a9e5-367d78845641" containerName="revoke-keys" Oct 31 22:04:26.365082 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.364975 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb51e128-721e-43ec-a9e5-367d78845641" containerName="revoke-keys" Oct 31 22:04:26.365205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.365191 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb51e128-721e-43ec-a9e5-367d78845641" containerName="revoke-keys" Oct 31 22:04:26.370202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.370181 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.387779 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.387751 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq"] Oct 31 22:04:26.393803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.393783 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-465d0a483ad1\"" Oct 31 22:04:26.446957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.446922 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-provider-credentials\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.447234 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.447213 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-plugins-config-volume\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.447473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.447450 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-routing-overlay-volume\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.447729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.447709 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87nzh\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-kube-api-access-87nzh\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.549483 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.549444 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-provider-credentials\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.549675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.549496 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-plugins-config-volume\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.549675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.549528 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-routing-overlay-volume\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.549675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.549608 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87nzh\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-kube-api-access-87nzh\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.550195 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.550169 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-routing-overlay-volume\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.550294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.550217 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-plugins-config-volume\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.552057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.552031 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-provider-credentials\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.573049 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.573023 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87nzh\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-kube-api-access-87nzh\") pod \"payload-processing-external-model-58cdbd54fd-wbdtq\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.684046 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.683961 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:26.987630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:26.987597 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq"] Oct 31 22:04:26.987935 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:04:26.987900 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe7893c_9dc9_43ef_8a4f_b03c7e1a013a.slice/crio-e44918c9b8c04bb421e94cda0d73c696239f9421b14f99a9d1dabe69317dbda9 WatchSource:0}: Error finding container e44918c9b8c04bb421e94cda0d73c696239f9421b14f99a9d1dabe69317dbda9: Status 404 returned error can't find the container with id e44918c9b8c04bb421e94cda0d73c696239f9421b14f99a9d1dabe69317dbda9 Oct 31 22:04:27.007646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:27.007617 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" event={"ID":"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a","Type":"ContainerStarted","Data":"e44918c9b8c04bb421e94cda0d73c696239f9421b14f99a9d1dabe69317dbda9"} Oct 31 22:04:28.016146 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.016109 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" event={"ID":"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a","Type":"ContainerStarted","Data":"b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245"} Oct 31 22:04:28.016664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.016219 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:28.059875 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.059830 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" podStartSLOduration=2.059817501 podStartE2EDuration="2.059817501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:04:28.056900237 +0000 UTC m=+2908.182007493" watchObservedRunningTime="2026-10-31 22:04:28.059817501 +0000 UTC m=+2908.184924731" Oct 31 22:04:28.165166 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.165122 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" podUID="ba74cf99-5fb7-4c8e-8f5e-737afa143a09" containerName="payload-processing" containerID="cri-o://4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915" gracePeriod=25 Oct 31 22:04:28.481665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.481547 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:04:28.602936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.602843 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-plugins-config-volume\") pod \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " Oct 31 22:04:28.602936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.602905 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-kube-api-access-76746\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-kube-api-access-76746\") pod \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " Oct 31 22:04:28.602936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.602930 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-provider-credentials\") pod \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " Oct 31 22:04:28.603348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.603328 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-routing-overlay-volume\") pod \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\" (UID: \"ba74cf99-5fb7-4c8e-8f5e-737afa143a09\") " Oct 31 22:04:28.603441 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.603350 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-plugins-config-volume" pod "ba74cf99-5fb7-4c8e-8f5e-737afa143a09" (UID: "ba74cf99-5fb7-4c8e-8f5e-737afa143a09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:04:28.603774 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.603745 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-routing-overlay-volume" pod "ba74cf99-5fb7-4c8e-8f5e-737afa143a09" (UID: "ba74cf99-5fb7-4c8e-8f5e-737afa143a09"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:04:28.604053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.604030 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:28.604136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.604124 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:28.605317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.605276 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-provider-credentials" pod "ba74cf99-5fb7-4c8e-8f5e-737afa143a09" (UID: "ba74cf99-5fb7-4c8e-8f5e-737afa143a09"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:04:28.605317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.605289 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-kube-api-access-76746" pod "ba74cf99-5fb7-4c8e-8f5e-737afa143a09" (UID: "ba74cf99-5fb7-4c8e-8f5e-737afa143a09"). InnerVolumeSpecName "kube-api-access-76746". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:04:28.706051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.706017 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76746\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-kube-api-access-76746\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:28.706051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:28.706044 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba74cf99-5fb7-4c8e-8f5e-737afa143a09-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:29.029828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.029789 4294 generic.go:356] "Generic (PLEG): container finished" podID="ba74cf99-5fb7-4c8e-8f5e-737afa143a09" containerID="4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915" exitCode=0 Oct 31 22:04:29.030355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.029862 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" Oct 31 22:04:29.030355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.029869 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" event={"ID":"ba74cf99-5fb7-4c8e-8f5e-737afa143a09","Type":"ContainerDied","Data":"4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915"} Oct 31 22:04:29.030355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.029937 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs" event={"ID":"ba74cf99-5fb7-4c8e-8f5e-737afa143a09","Type":"ContainerDied","Data":"495a17b62ee228184378a2d175efe9a32ce4826505e1130fb16578e8ca8a1c93"} Oct 31 22:04:29.030355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.029970 4294 scope.go:122] "RemoveContainer" containerID="4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915" Oct 31 22:04:29.043024 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.042998 4294 scope.go:122] "RemoveContainer" containerID="4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915" Oct 31 22:04:29.043385 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:04:29.043365 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915\": container with ID starting with 4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915 not found: ID does not exist" containerID="4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915" Oct 31 22:04:29.043475 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.043397 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915"} err="failed to get container status \"4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915\": rpc error: code = NotFound desc = could not find container \"4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915\": container with ID starting with 4b8c59708e7428c7a27c4b52c5e84c3f9d62fbcf49e04450587dd4cf1c333915 not found: ID does not exist" Oct 31 22:04:29.086691 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.086659 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs"] Oct 31 22:04:29.097077 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.097044 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-6dbdngxcs"] Oct 31 22:04:29.988185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:29.988150 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba74cf99-5fb7-4c8e-8f5e-737afa143a09" path="/var/lib/kubelet/pods/ba74cf99-5fb7-4c8e-8f5e-737afa143a09/volumes" Oct 31 22:04:31.369248 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:04:31.369207 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:04:32.051093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:32.051063 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z"] Oct 31 22:04:32.062109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:32.062055 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92d20848-nb26z"] Oct 31 22:04:33.985611 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:33.985578 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3a010a-9bdf-4755-bdb1-aa19feddd7c8" path="/var/lib/kubelet/pods/fe3a010a-9bdf-4755-bdb1-aa19feddd7c8/volumes" Oct 31 22:04:39.033805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:39.033775 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:04:39.110878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:39.110842 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r"] Oct 31 22:04:44.112200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.112113 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" podUID="84b487b2-55ee-40db-a33a-f22ad66c660c" containerName="payload-processing" containerID="cri-o://963d5adbfd2b68f60434838682c7b3796634a3e96b43664088053d26e7519191" gracePeriod=25 Oct 31 22:04:44.288271 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.288234 4294 generic.go:356] "Generic (PLEG): container finished" podID="84b487b2-55ee-40db-a33a-f22ad66c660c" containerID="963d5adbfd2b68f60434838682c7b3796634a3e96b43664088053d26e7519191" exitCode=0 Oct 31 22:04:44.288450 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.288284 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" event={"ID":"84b487b2-55ee-40db-a33a-f22ad66c660c","Type":"ContainerDied","Data":"963d5adbfd2b68f60434838682c7b3796634a3e96b43664088053d26e7519191"} Oct 31 22:04:44.381700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.381636 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:04:44.403877 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.403849 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-plugins-config-volume\") pod \"84b487b2-55ee-40db-a33a-f22ad66c660c\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " Oct 31 22:04:44.404025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.403940 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-kube-api-access-tpcv8\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-kube-api-access-tpcv8\") pod \"84b487b2-55ee-40db-a33a-f22ad66c660c\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " Oct 31 22:04:44.404025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.403965 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-provider-credentials\") pod \"84b487b2-55ee-40db-a33a-f22ad66c660c\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " Oct 31 22:04:44.404160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.404107 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-routing-overlay-volume\") pod \"84b487b2-55ee-40db-a33a-f22ad66c660c\" (UID: \"84b487b2-55ee-40db-a33a-f22ad66c660c\") " Oct 31 22:04:44.404281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.404258 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-plugins-config-volume" pod "84b487b2-55ee-40db-a33a-f22ad66c660c" (UID: "84b487b2-55ee-40db-a33a-f22ad66c660c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:04:44.404491 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.404467 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-routing-overlay-volume" pod "84b487b2-55ee-40db-a33a-f22ad66c660c" (UID: "84b487b2-55ee-40db-a33a-f22ad66c660c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:04:44.404737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.404718 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:44.404737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.404738 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b487b2-55ee-40db-a33a-f22ad66c660c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:44.406066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.406042 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-kube-api-access-tpcv8" pod "84b487b2-55ee-40db-a33a-f22ad66c660c" (UID: "84b487b2-55ee-40db-a33a-f22ad66c660c"). InnerVolumeSpecName "kube-api-access-tpcv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:04:44.406200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.406181 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-provider-credentials" pod "84b487b2-55ee-40db-a33a-f22ad66c660c" (UID: "84b487b2-55ee-40db-a33a-f22ad66c660c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:04:44.505831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.505802 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcv8\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-kube-api-access-tpcv8\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:44.505831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:44.505830 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84b487b2-55ee-40db-a33a-f22ad66c660c-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:04:45.297664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:45.297635 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" Oct 31 22:04:45.298086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:45.297667 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r" event={"ID":"84b487b2-55ee-40db-a33a-f22ad66c660c","Type":"ContainerDied","Data":"11cdc26d72c04f571e058b6187a85db7c67e93005afe61280e6ce5e6de90109d"} Oct 31 22:04:45.298086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:45.297731 4294 scope.go:122] "RemoveContainer" containerID="963d5adbfd2b68f60434838682c7b3796634a3e96b43664088053d26e7519191" Oct 31 22:04:45.351036 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:45.351008 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r"] Oct 31 22:04:45.363000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:45.362976 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-67ddf5bb68-5ss2r"] Oct 31 22:04:45.985601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:04:45.985567 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b487b2-55ee-40db-a33a-f22ad66c660c" path="/var/lib/kubelet/pods/84b487b2-55ee-40db-a33a-f22ad66c660c/volumes" Oct 31 22:05:01.310297 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:05:01.310262 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:05:02.701793 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:02.701755 4294 scope.go:122] "RemoveContainer" containerID="2f908665369db95ff9af4fc735e5aca38fc962aa4abc6520f28114ff263b86a0" Oct 31 22:05:02.711694 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:02.711674 4294 scope.go:122] "RemoveContainer" containerID="2c5668e6a0f5a2efcbcf7a4c52a09698dc7fc78e9838a76159699ee7b25e117e" Oct 31 22:05:02.720476 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:02.720458 4294 scope.go:122] "RemoveContainer" containerID="28b042543fc93643328cdee1a6be5a96add2a134e632586a24d1a3686514ba03" Oct 31 22:05:04.034411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:04.034383 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s"] Oct 31 22:05:04.045928 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:04.045904 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31b990fe-wsf8s"] Oct 31 22:05:05.986942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:05.986911 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="609de17a-3818-49a4-a0c6-3edc99e4337d" path="/var/lib/kubelet/pods/609de17a-3818-49a4-a0c6-3edc99e4337d/volumes" Oct 31 22:05:15.033516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:15.033478 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq"] Oct 31 22:05:15.044993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:15.044962 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f715b9dd-bgljq"] Oct 31 22:05:15.991667 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:15.991630 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a44c138-1a2c-4bb9-b57c-32155db4050e" path="/var/lib/kubelet/pods/0a44c138-1a2c-4bb9-b57c-32155db4050e/volumes" Oct 31 22:05:23.688507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.688476 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z"] Oct 31 22:05:23.689074 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.689060 4294 cpu_manager.go:414] "containerMap: removing container" podUID="84b487b2-55ee-40db-a33a-f22ad66c660c" containerName="payload-processing" Oct 31 22:05:23.689125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.689075 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b487b2-55ee-40db-a33a-f22ad66c660c" containerName="payload-processing" Oct 31 22:05:23.689125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.689106 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ba74cf99-5fb7-4c8e-8f5e-737afa143a09" containerName="payload-processing" Oct 31 22:05:23.689125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.689111 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba74cf99-5fb7-4c8e-8f5e-737afa143a09" containerName="payload-processing" Oct 31 22:05:23.689223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.689214 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b487b2-55ee-40db-a33a-f22ad66c660c" containerName="payload-processing" Oct 31 22:05:23.689258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.689228 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba74cf99-5fb7-4c8e-8f5e-737afa143a09" containerName="payload-processing" Oct 31 22:05:23.694133 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.694115 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.712346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.712293 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z"] Oct 31 22:05:23.718764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.718658 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"kube-root-ca.crt\"" Oct 31 22:05:23.718764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.718677 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-56a6d4\"" Oct 31 22:05:23.718764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.718658 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"agc-cred-96a32c16efa4\"" Oct 31 22:05:23.718764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.718681 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"routing-overlay\"" Oct 31 22:05:23.718764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.718739 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"openshift-service-ca.crt\"" Oct 31 22:05:23.721292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.721271 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-e2e--34b55c0e-dockercfg-7d7kg\"" Oct 31 22:05:23.776773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.776746 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.776949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.776926 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.776999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.776988 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnbv\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-kube-api-access-cvnbv\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.777070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.777054 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.906326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.884331 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.906326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.884431 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnbv\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-kube-api-access-cvnbv\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.906326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.884499 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.906326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.884585 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.906326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.885677 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.906326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.887027 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.922322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.918912 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:23.932568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:23.932508 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnbv\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-kube-api-access-cvnbv\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:24.008788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:24.008752 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:24.312635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:24.312606 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z"] Oct 31 22:05:24.313581 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:05:24.313552 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bd02d6_531f_49ac_aaa6_bd6c127e686f.slice/crio-e68ef40fe4638a5c2322172231ab6634c96c67e8f2c645ed6c3d3e635495a811 WatchSource:0}: Error finding container e68ef40fe4638a5c2322172231ab6634c96c67e8f2c645ed6c3d3e635495a811: Status 404 returned error can't find the container with id e68ef40fe4638a5c2322172231ab6634c96c67e8f2c645ed6c3d3e635495a811 Oct 31 22:05:24.613799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:24.613704 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" event={"ID":"84bd02d6-531f-49ac-aaa6-bd6c127e686f","Type":"ContainerStarted","Data":"eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f"} Oct 31 22:05:24.613799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:24.613761 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" event={"ID":"84bd02d6-531f-49ac-aaa6-bd6c127e686f","Type":"ContainerStarted","Data":"e68ef40fe4638a5c2322172231ab6634c96c67e8f2c645ed6c3d3e635495a811"} Oct 31 22:05:24.614028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:24.613853 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:05:24.658397 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:24.658345 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" podStartSLOduration=1.6583280999999999 podStartE2EDuration="1.6583281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:05:24.654667534 +0000 UTC m=+2964.779774766" watchObservedRunningTime="2026-10-31 22:05:24.6583281 +0000 UTC m=+2964.783435333" Oct 31 22:05:27.034096 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:27.034063 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z"] Oct 31 22:05:27.046472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:27.046446 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e2967b-kxw2z"] Oct 31 22:05:27.986942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:27.986914 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e808011d-7e69-4029-b11f-d22c647262e3" path="/var/lib/kubelet/pods/e808011d-7e69-4029-b11f-d22c647262e3/volumes" Oct 31 22:05:31.311151 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:05:31.311116 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:05:35.624560 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:05:35.624529 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:06:00.549334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:00.549289 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:06:00.564838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:00.564814 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:06:01.310148 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:06:01.310116 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:06:02.767981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:02.767949 4294 scope.go:122] "RemoveContainer" containerID="9b81c58bc412de303391b266858128cb098b6f31736243c7aeda876d5b4b90f4" Oct 31 22:06:02.778171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:02.778151 4294 scope.go:122] "RemoveContainer" containerID="f33291453a48c51f597d7663370f7618162b41f4406f53c8d8771216286df051" Oct 31 22:06:02.787561 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:02.787539 4294 scope.go:122] "RemoveContainer" containerID="2339745ff972b92365fee11d89b56699c5268420b210ce1974e65b9dc35beacf" Oct 31 22:06:02.796211 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:02.796189 4294 scope.go:122] "RemoveContainer" containerID="28f0fad6777f0a00f49f7390aaf305952c6bf67210201c11d468879d6c1f88e4" Oct 31 22:06:13.055239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:13.055158 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv"] Oct 31 22:06:13.066386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:13.066360 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737cc848-crdsv"] Oct 31 22:06:13.985922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:13.985876 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519e661e-03ce-4b9c-87b2-21d26ce2d3fb" path="/var/lib/kubelet/pods/519e661e-03ce-4b9c-87b2-21d26ce2d3fb/volumes" Oct 31 22:06:25.032459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:25.032426 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv"] Oct 31 22:06:25.043863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:25.043839 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da1c079c-vkjzv"] Oct 31 22:06:25.985972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:25.985938 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c744edf9-814d-46d1-aeda-ae19ac026ebc" path="/var/lib/kubelet/pods/c744edf9-814d-46d1-aeda-ae19ac026ebc/volumes" Oct 31 22:06:28.032111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:28.032073 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp"] Oct 31 22:06:28.044100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:28.044075 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f11a5a9f-svwcp"] Oct 31 22:06:28.046938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:28.046920 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7"] Oct 31 22:06:28.049784 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:28.049766 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f1d87a8-8hvf7"] Oct 31 22:06:29.988030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:29.987995 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3f125d6-ccbf-4cb0-8e6d-e37329991737" path="/var/lib/kubelet/pods/c3f125d6-ccbf-4cb0-8e6d-e37329991737/volumes" Oct 31 22:06:29.988671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:29.988650 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83c35d0-8de3-4def-a6ea-032dbd5362d7" path="/var/lib/kubelet/pods/e83c35d0-8de3-4def-a6ea-032dbd5362d7/volumes" Oct 31 22:06:31.312388 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:06:31.312352 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:06:39.051950 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:39.051914 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9"] Oct 31 22:06:39.065687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:39.065655 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85171cb4-m5zs9"] Oct 31 22:06:39.986884 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:39.986846 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="044f0321-8883-432a-a4be-273877d04cc7" path="/var/lib/kubelet/pods/044f0321-8883-432a-a4be-273877d04cc7/volumes" Oct 31 22:06:42.463253 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.463210 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq"] Oct 31 22:06:42.467954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.467930 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.486430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.486398 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq"] Oct 31 22:06:42.493071 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.493038 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:06:42.493071 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.493057 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:06:42.507616 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.507588 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.507812 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.507786 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.507948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.507929 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp6lx\" (UniqueName: \"kubernetes.io/projected/76056de0-2080-4ff1-87d3-4aa034d6bd76-kube-api-access-xp6lx\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.563100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.563061 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx"] Oct 31 22:06:42.567429 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.567409 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.583915 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.583888 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4a73b1\"/\"agc-cred-d10d6425f537\"" Oct 31 22:06:42.585126 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.585103 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx"] Oct 31 22:06:42.608860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.608825 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.609052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.608889 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.609052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.608928 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r4xf\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-kube-api-access-7r4xf\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.609052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.608996 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.609232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.609143 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.609232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.609224 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp6lx\" (UniqueName: \"kubernetes.io/projected/76056de0-2080-4ff1-87d3-4aa034d6bd76-kube-api-access-xp6lx\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.609391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.609278 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.609734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.609714 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.611644 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.611616 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.633335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.633277 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp6lx\" (UniqueName: \"kubernetes.io/projected/76056de0-2080-4ff1-87d3-4aa034d6bd76-kube-api-access-xp6lx\") pod \"maas-api-revoke-sub-67d715ad-kjcdq\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.710616 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.710577 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r4xf\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-kube-api-access-7r4xf\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.710825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.710728 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.710895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.710821 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.711016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.710989 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.711642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.711612 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.711752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.711741 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.713822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.713762 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.734229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.734198 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r4xf\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-kube-api-access-7r4xf\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:42.782200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.782167 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:42.882716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:42.882684 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:43.089323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:43.089282 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq"] Oct 31 22:06:43.090136 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:06:43.090106 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76056de0_2080_4ff1_87d3_4aa034d6bd76.slice/crio-02871f35f0ab8916cc2b3a2a1ab6f02f090ee7ff551790599ffffed74593d30f WatchSource:0}: Error finding container 02871f35f0ab8916cc2b3a2a1ab6f02f090ee7ff551790599ffffed74593d30f: Status 404 returned error can't find the container with id 02871f35f0ab8916cc2b3a2a1ab6f02f090ee7ff551790599ffffed74593d30f Oct 31 22:06:43.195978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:43.195948 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx"] Oct 31 22:06:43.197237 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:06:43.197209 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff335783_0ce6_410b_b80e_209988d7ebea.slice/crio-a0617c001145ec63c337cc591655ee4758eb62274e0d7829d933ed14f7ca8f7f WatchSource:0}: Error finding container a0617c001145ec63c337cc591655ee4758eb62274e0d7829d933ed14f7ca8f7f: Status 404 returned error can't find the container with id a0617c001145ec63c337cc591655ee4758eb62274e0d7829d933ed14f7ca8f7f Oct 31 22:06:43.295344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:43.295288 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" event={"ID":"76056de0-2080-4ff1-87d3-4aa034d6bd76","Type":"ContainerStarted","Data":"02871f35f0ab8916cc2b3a2a1ab6f02f090ee7ff551790599ffffed74593d30f"} Oct 31 22:06:43.297020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:43.296993 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" event={"ID":"ff335783-0ce6-410b-b80e-209988d7ebea","Type":"ContainerStarted","Data":"a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04"} Oct 31 22:06:43.297153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:43.297038 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" event={"ID":"ff335783-0ce6-410b-b80e-209988d7ebea","Type":"ContainerStarted","Data":"a0617c001145ec63c337cc591655ee4758eb62274e0d7829d933ed14f7ca8f7f"} Oct 31 22:06:43.297153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:43.297091 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:43.343647 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:43.343559 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" podStartSLOduration=1.3435450709999999 podStartE2EDuration="1.343545071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:06:43.339912844 +0000 UTC m=+3043.465020074" watchObservedRunningTime="2026-10-31 22:06:43.343545071 +0000 UTC m=+3043.468652301" Oct 31 22:06:44.306565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:44.306517 4294 generic.go:356] "Generic (PLEG): container finished" podID="76056de0-2080-4ff1-87d3-4aa034d6bd76" containerID="8a3114c4bd3efbc04be02fa7430f9f5f3139d886cb2626d3ece85d79161b4a58" exitCode=0 Oct 31 22:06:44.307002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:44.306557 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" event={"ID":"76056de0-2080-4ff1-87d3-4aa034d6bd76","Type":"ContainerDied","Data":"8a3114c4bd3efbc04be02fa7430f9f5f3139d886cb2626d3ece85d79161b4a58"} Oct 31 22:06:45.489945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.489920 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:45.556077 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.556041 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76056de0-2080-4ff1-87d3-4aa034d6bd76-kube-api-access-xp6lx\" (UniqueName: \"kubernetes.io/projected/76056de0-2080-4ff1-87d3-4aa034d6bd76-kube-api-access-xp6lx\") pod \"76056de0-2080-4ff1-87d3-4aa034d6bd76\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " Oct 31 22:06:45.556268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.556103 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-xks-ca\") pod \"76056de0-2080-4ff1-87d3-4aa034d6bd76\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " Oct 31 22:06:45.556268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.556247 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-openshift-ca\") pod \"76056de0-2080-4ff1-87d3-4aa034d6bd76\" (UID: \"76056de0-2080-4ff1-87d3-4aa034d6bd76\") " Oct 31 22:06:45.556819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.556788 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-openshift-ca" pod "76056de0-2080-4ff1-87d3-4aa034d6bd76" (UID: "76056de0-2080-4ff1-87d3-4aa034d6bd76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:06:45.557065 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.557038 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:45.558762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.558737 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-xks-ca" pod "76056de0-2080-4ff1-87d3-4aa034d6bd76" (UID: "76056de0-2080-4ff1-87d3-4aa034d6bd76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:06:45.558865 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.558770 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76056de0-2080-4ff1-87d3-4aa034d6bd76-kube-api-access-xp6lx" pod "76056de0-2080-4ff1-87d3-4aa034d6bd76" (UID: "76056de0-2080-4ff1-87d3-4aa034d6bd76"). InnerVolumeSpecName "kube-api-access-xp6lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:06:45.658464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.658431 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp6lx\" (UniqueName: \"kubernetes.io/projected/76056de0-2080-4ff1-87d3-4aa034d6bd76-kube-api-access-xp6lx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:45.658464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:45.658463 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76056de0-2080-4ff1-87d3-4aa034d6bd76-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:46.323945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:46.323859 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" Oct 31 22:06:46.323945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:46.323880 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq" event={"ID":"76056de0-2080-4ff1-87d3-4aa034d6bd76","Type":"ContainerDied","Data":"02871f35f0ab8916cc2b3a2a1ab6f02f090ee7ff551790599ffffed74593d30f"} Oct 31 22:06:46.323945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:46.323935 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02871f35f0ab8916cc2b3a2a1ab6f02f090ee7ff551790599ffffed74593d30f" Oct 31 22:06:53.032822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:53.032785 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6"] Oct 31 22:06:53.044229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:53.044198 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-498c255c-trkh6"] Oct 31 22:06:53.690036 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:53.689999 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z"] Oct 31 22:06:53.993748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:53.993712 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac2b563-6516-498d-959a-6ba52742b4a1" path="/var/lib/kubelet/pods/2ac2b563-6516-498d-959a-6ba52742b4a1/volumes" Oct 31 22:06:54.310883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.310801 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:06:54.342413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.342382 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs"] Oct 31 22:06:54.343011 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.342997 4294 cpu_manager.go:414] "containerMap: removing container" podUID="76056de0-2080-4ff1-87d3-4aa034d6bd76" containerName="revoke-keys" Oct 31 22:06:54.343063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.343012 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76056de0-2080-4ff1-87d3-4aa034d6bd76" containerName="revoke-keys" Oct 31 22:06:54.343154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.343143 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="76056de0-2080-4ff1-87d3-4aa034d6bd76" containerName="revoke-keys" Oct 31 22:06:54.347908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.347890 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.362010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.361979 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs"] Oct 31 22:06:54.362728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.362704 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:06:54.364434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.364412 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:06:54.393102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.392786 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc"] Oct 31 22:06:54.487742 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.487697 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.487962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.487892 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbsxq\" (UniqueName: \"kubernetes.io/projected/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-kube-api-access-gbsxq\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.488102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.488084 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.601191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.597362 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbsxq\" (UniqueName: \"kubernetes.io/projected/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-kube-api-access-gbsxq\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.601191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.597593 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.601191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.597712 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.601191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.599803 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.604977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.604948 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.621285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.621251 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbsxq\" (UniqueName: \"kubernetes.io/projected/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-kube-api-access-gbsxq\") pod \"maas-api-revoke-sub-d2705900-fq7rs\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.663378 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.663349 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:54.968393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:54.968354 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs"] Oct 31 22:06:54.969691 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:06:54.969654 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8dcd09_22d9_41b5_8b29_7fa9ae6f4f1d.slice/crio-3e0548cbc0810467cce6899b12dffecf54875e38bfeece3f88820f8a47f1398a WatchSource:0}: Error finding container 3e0548cbc0810467cce6899b12dffecf54875e38bfeece3f88820f8a47f1398a: Status 404 returned error can't find the container with id 3e0548cbc0810467cce6899b12dffecf54875e38bfeece3f88820f8a47f1398a Oct 31 22:06:55.401730 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:55.401687 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" event={"ID":"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d","Type":"ContainerStarted","Data":"f9230f060a77e432f3f421215d5855262f3add1c10db87ffd1a90624e8fc8b62"} Oct 31 22:06:55.402132 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:55.401749 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" event={"ID":"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d","Type":"ContainerStarted","Data":"3e0548cbc0810467cce6899b12dffecf54875e38bfeece3f88820f8a47f1398a"} Oct 31 22:06:55.448214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:55.448146 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" podStartSLOduration=1.448125625 podStartE2EDuration="1.448125625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:06:55.442235733 +0000 UTC m=+3055.567342966" watchObservedRunningTime="2026-10-31 22:06:55.448125625 +0000 UTC m=+3055.573232857" Oct 31 22:06:55.517841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:55.517802 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq"] Oct 31 22:06:56.444588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:56.441718 4294 generic.go:356] "Generic (PLEG): container finished" podID="0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" containerID="f9230f060a77e432f3f421215d5855262f3add1c10db87ffd1a90624e8fc8b62" exitCode=0 Oct 31 22:06:56.444588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:56.441769 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" event={"ID":"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d","Type":"ContainerDied","Data":"f9230f060a77e432f3f421215d5855262f3add1c10db87ffd1a90624e8fc8b62"} Oct 31 22:06:57.607967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.607939 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:57.668712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.668683 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-kube-api-access-gbsxq\" (UniqueName: \"kubernetes.io/projected/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-kube-api-access-gbsxq\") pod \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " Oct 31 22:06:57.668908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.668775 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-xks-ca\") pod \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " Oct 31 22:06:57.669093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.669059 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-openshift-ca\") pod \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\" (UID: \"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d\") " Oct 31 22:06:57.669506 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.669477 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-openshift-ca" pod "0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" (UID: "0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:06:57.669761 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.669745 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:57.671425 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.671401 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-kube-api-access-gbsxq" pod "0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" (UID: "0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d"). InnerVolumeSpecName "kube-api-access-gbsxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:06:57.671650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.671623 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-xks-ca" pod "0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" (UID: "0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:06:57.771709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.771669 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:57.771709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:57.771704 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbsxq\" (UniqueName: \"kubernetes.io/projected/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d-kube-api-access-gbsxq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:58.459363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:58.459332 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" Oct 31 22:06:58.459363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:58.459352 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs" event={"ID":"0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d","Type":"ContainerDied","Data":"3e0548cbc0810467cce6899b12dffecf54875e38bfeece3f88820f8a47f1398a"} Oct 31 22:06:58.459630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:58.459410 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e0548cbc0810467cce6899b12dffecf54875e38bfeece3f88820f8a47f1398a" Oct 31 22:06:58.691021 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:58.690983 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" podUID="84bd02d6-531f-49ac-aaa6-bd6c127e686f" containerName="payload-processing" containerID="cri-o://eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f" gracePeriod=25 Oct 31 22:06:58.975680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:58.975608 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:06:59.089440 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.089403 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-kube-api-access-cvnbv\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-kube-api-access-cvnbv\") pod \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " Oct 31 22:06:59.089668 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.089654 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-plugins-config-volume\") pod \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " Oct 31 22:06:59.089733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.089704 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-provider-credentials\") pod \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " Oct 31 22:06:59.089792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.089770 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-routing-overlay-volume\") pod \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\" (UID: \"84bd02d6-531f-49ac-aaa6-bd6c127e686f\") " Oct 31 22:06:59.090156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.090109 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-plugins-config-volume" pod "84bd02d6-531f-49ac-aaa6-bd6c127e686f" (UID: "84bd02d6-531f-49ac-aaa6-bd6c127e686f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:06:59.090296 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.090168 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-routing-overlay-volume" pod "84bd02d6-531f-49ac-aaa6-bd6c127e686f" (UID: "84bd02d6-531f-49ac-aaa6-bd6c127e686f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:06:59.090475 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.090451 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.090571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.090477 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bd02d6-531f-49ac-aaa6-bd6c127e686f-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.091739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.091713 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-kube-api-access-cvnbv" pod "84bd02d6-531f-49ac-aaa6-bd6c127e686f" (UID: "84bd02d6-531f-49ac-aaa6-bd6c127e686f"). InnerVolumeSpecName "kube-api-access-cvnbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:06:59.091913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.091891 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-provider-credentials" pod "84bd02d6-531f-49ac-aaa6-bd6c127e686f" (UID: "84bd02d6-531f-49ac-aaa6-bd6c127e686f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:06:59.192317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.192277 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.192488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.192338 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnbv\" (UniqueName: \"kubernetes.io/projected/84bd02d6-531f-49ac-aaa6-bd6c127e686f-kube-api-access-cvnbv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.393829 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.393773 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" podUID="59404ca5-f2a4-48a4-9098-5b0c65814957" containerName="payload-processing" containerID="cri-o://49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc" gracePeriod=25 Oct 31 22:06:59.468452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.468420 4294 generic.go:356] "Generic (PLEG): container finished" podID="84bd02d6-531f-49ac-aaa6-bd6c127e686f" containerID="eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f" exitCode=0 Oct 31 22:06:59.468642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.468476 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" Oct 31 22:06:59.468642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.468496 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" event={"ID":"84bd02d6-531f-49ac-aaa6-bd6c127e686f","Type":"ContainerDied","Data":"eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f"} Oct 31 22:06:59.468642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.468548 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z" event={"ID":"84bd02d6-531f-49ac-aaa6-bd6c127e686f","Type":"ContainerDied","Data":"e68ef40fe4638a5c2322172231ab6634c96c67e8f2c645ed6c3d3e635495a811"} Oct 31 22:06:59.468642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.468569 4294 scope.go:122] "RemoveContainer" containerID="eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f" Oct 31 22:06:59.507073 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.507050 4294 scope.go:122] "RemoveContainer" containerID="eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f" Oct 31 22:06:59.507442 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:06:59.507417 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f\": container with ID starting with eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f not found: ID does not exist" containerID="eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f" Oct 31 22:06:59.507534 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.507452 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f"} err="failed to get container status \"eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f\": rpc error: code = NotFound desc = could not find container \"eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f\": container with ID starting with eb22c0c21282bec346043835a810bb0527b84561117350f9fcec07c97187045f not found: ID does not exist" Oct 31 22:06:59.522667 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.522636 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z"] Oct 31 22:06:59.534464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.534434 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-5787wnp9z"] Oct 31 22:06:59.648654 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.648599 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:06:59.699030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.698999 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-provider-credentials\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-provider-credentials\") pod \"59404ca5-f2a4-48a4-9098-5b0c65814957\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " Oct 31 22:06:59.699030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.699040 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-kube-api-access-jmxfp\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-kube-api-access-jmxfp\") pod \"59404ca5-f2a4-48a4-9098-5b0c65814957\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " Oct 31 22:06:59.699652 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.699257 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-routing-overlay-volume\") pod \"59404ca5-f2a4-48a4-9098-5b0c65814957\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " Oct 31 22:06:59.699652 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.699280 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-plugins-config-volume\") pod \"59404ca5-f2a4-48a4-9098-5b0c65814957\" (UID: \"59404ca5-f2a4-48a4-9098-5b0c65814957\") " Oct 31 22:06:59.699769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.699705 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-plugins-config-volume" pod "59404ca5-f2a4-48a4-9098-5b0c65814957" (UID: "59404ca5-f2a4-48a4-9098-5b0c65814957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:06:59.699920 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.699899 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.700244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.700226 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-routing-overlay-volume" pod "59404ca5-f2a4-48a4-9098-5b0c65814957" (UID: "59404ca5-f2a4-48a4-9098-5b0c65814957"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:06:59.704550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.704521 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-provider-credentials" pod "59404ca5-f2a4-48a4-9098-5b0c65814957" (UID: "59404ca5-f2a4-48a4-9098-5b0c65814957"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:06:59.704674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.704563 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-kube-api-access-jmxfp" pod "59404ca5-f2a4-48a4-9098-5b0c65814957" (UID: "59404ca5-f2a4-48a4-9098-5b0c65814957"). InnerVolumeSpecName "kube-api-access-jmxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:06:59.801627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.801597 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/59404ca5-f2a4-48a4-9098-5b0c65814957-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.801627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.801624 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.801627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.801634 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmxfp\" (UniqueName: \"kubernetes.io/projected/59404ca5-f2a4-48a4-9098-5b0c65814957-kube-api-access-jmxfp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:06:59.987536 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:06:59.987500 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bd02d6-531f-49ac-aaa6-bd6c127e686f" path="/var/lib/kubelet/pods/84bd02d6-531f-49ac-aaa6-bd6c127e686f/volumes" Oct 31 22:07:00.478006 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.477927 4294 generic.go:356] "Generic (PLEG): container finished" podID="59404ca5-f2a4-48a4-9098-5b0c65814957" containerID="49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc" exitCode=0 Oct 31 22:07:00.478006 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.477978 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" Oct 31 22:07:00.478214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.478009 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" event={"ID":"59404ca5-f2a4-48a4-9098-5b0c65814957","Type":"ContainerDied","Data":"49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc"} Oct 31 22:07:00.478214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.478062 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc" event={"ID":"59404ca5-f2a4-48a4-9098-5b0c65814957","Type":"ContainerDied","Data":"6e690b368b8cfae669bc4a9478febd99f087813e30ad119e833eb054c444e0d2"} Oct 31 22:07:00.478214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.478083 4294 scope.go:122] "RemoveContainer" containerID="49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc" Oct 31 22:07:00.488196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.488171 4294 scope.go:122] "RemoveContainer" containerID="49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc" Oct 31 22:07:00.488495 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:07:00.488472 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc\": container with ID starting with 49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc not found: ID does not exist" containerID="49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc" Oct 31 22:07:00.488558 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.488507 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc"} err="failed to get container status \"49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc\": rpc error: code = NotFound desc = could not find container \"49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc\": container with ID starting with 49875baeaf714d481fb565dc77d627401ef432e64468ccf3f116529e315e6edc not found: ID does not exist" Oct 31 22:07:00.519046 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.518999 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" podUID="28024b1e-f5cd-48a3-b7ce-53b6f2403c29" containerName="payload-processing" containerID="cri-o://c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085" gracePeriod=25 Oct 31 22:07:00.531335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.531293 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc"] Oct 31 22:07:00.544854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.544825 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f66gsnhc"] Oct 31 22:07:00.797053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.797023 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:07:00.916853 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.916816 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-plugins-config-volume\") pod \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " Oct 31 22:07:00.917040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.917023 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-kube-api-access-4tml7\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-kube-api-access-4tml7\") pod \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " Oct 31 22:07:00.917087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.917053 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-provider-credentials\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-provider-credentials\") pod \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " Oct 31 22:07:00.917087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.917079 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-routing-overlay-volume\") pod \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\" (UID: \"28024b1e-f5cd-48a3-b7ce-53b6f2403c29\") " Oct 31 22:07:00.917236 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.917213 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-plugins-config-volume" pod "28024b1e-f5cd-48a3-b7ce-53b6f2403c29" (UID: "28024b1e-f5cd-48a3-b7ce-53b6f2403c29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:00.917541 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.917509 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-routing-overlay-volume" pod "28024b1e-f5cd-48a3-b7ce-53b6f2403c29" (UID: "28024b1e-f5cd-48a3-b7ce-53b6f2403c29"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:00.917712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.917692 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:00.917712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.917712 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:00.919233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.919211 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-provider-credentials" pod "28024b1e-f5cd-48a3-b7ce-53b6f2403c29" (UID: "28024b1e-f5cd-48a3-b7ce-53b6f2403c29"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:00.919356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:00.919254 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-kube-api-access-4tml7" pod "28024b1e-f5cd-48a3-b7ce-53b6f2403c29" (UID: "28024b1e-f5cd-48a3-b7ce-53b6f2403c29"). InnerVolumeSpecName "kube-api-access-4tml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:01.018777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.018691 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tml7\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-kube-api-access-4tml7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:01.018777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.018726 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/28024b1e-f5cd-48a3-b7ce-53b6f2403c29-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:01.329565 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:07:01.327100 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:07:01.488647 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.488616 4294 generic.go:356] "Generic (PLEG): container finished" podID="28024b1e-f5cd-48a3-b7ce-53b6f2403c29" containerID="c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085" exitCode=0 Oct 31 22:07:01.488846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.488662 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" Oct 31 22:07:01.488846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.488700 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" event={"ID":"28024b1e-f5cd-48a3-b7ce-53b6f2403c29","Type":"ContainerDied","Data":"c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085"} Oct 31 22:07:01.488846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.488755 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq" event={"ID":"28024b1e-f5cd-48a3-b7ce-53b6f2403c29","Type":"ContainerDied","Data":"bfd0c650e8e8940434d4105a12bd812bba49e4b843f3fe9e19ee6b3b5d4cff2f"} Oct 31 22:07:01.488846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.488783 4294 scope.go:122] "RemoveContainer" containerID="c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085" Oct 31 22:07:01.500475 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.500455 4294 scope.go:122] "RemoveContainer" containerID="c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085" Oct 31 22:07:01.500757 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:07:01.500736 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085\": container with ID starting with c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085 not found: ID does not exist" containerID="c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085" Oct 31 22:07:01.500834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.500766 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085"} err="failed to get container status \"c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085\": rpc error: code = NotFound desc = could not find container \"c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085\": container with ID starting with c49a1c7f39170f7b31927cc7eb43da8319d63ccfc1917bc505474478c2eb6085 not found: ID does not exist" Oct 31 22:07:01.541863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.541835 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq"] Oct 31 22:07:01.553288 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.553264 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-7bf5qhgcq"] Oct 31 22:07:01.985883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.985850 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28024b1e-f5cd-48a3-b7ce-53b6f2403c29" path="/var/lib/kubelet/pods/28024b1e-f5cd-48a3-b7ce-53b6f2403c29/volumes" Oct 31 22:07:01.986296 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:01.986270 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59404ca5-f2a4-48a4-9098-5b0c65814957" path="/var/lib/kubelet/pods/59404ca5-f2a4-48a4-9098-5b0c65814957/volumes" Oct 31 22:07:02.839644 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:02.839612 4294 scope.go:122] "RemoveContainer" containerID="a5077684715094f6d424da79b7fc18e959c6837abcb5d1a6c2bbf676fdcd60e0" Oct 31 22:07:02.849854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:02.849832 4294 scope.go:122] "RemoveContainer" containerID="6ba6aac779557d7250ee2e47d41262643aa872c09de8ea86fca5f51a7335edae" Oct 31 22:07:10.033482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:10.033453 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j"] Oct 31 22:07:10.045229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:10.045205 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a05f4ee3-7j97j"] Oct 31 22:07:11.985756 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:11.985720 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f16b1c1-5fd5-4e3e-b9bc-9703585d0492" path="/var/lib/kubelet/pods/8f16b1c1-5fd5-4e3e-b9bc-9703585d0492/volumes" Oct 31 22:07:16.339137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.339096 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm"] Oct 31 22:07:16.339923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.339901 4294 cpu_manager.go:414] "containerMap: removing container" podUID="28024b1e-f5cd-48a3-b7ce-53b6f2403c29" containerName="payload-processing" Oct 31 22:07:16.340030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.339925 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28024b1e-f5cd-48a3-b7ce-53b6f2403c29" containerName="payload-processing" Oct 31 22:07:16.340030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.339945 4294 cpu_manager.go:414] "containerMap: removing container" podUID="84bd02d6-531f-49ac-aaa6-bd6c127e686f" containerName="payload-processing" Oct 31 22:07:16.340030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.339951 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bd02d6-531f-49ac-aaa6-bd6c127e686f" containerName="payload-processing" Oct 31 22:07:16.340030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.339962 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" containerName="revoke-keys" Oct 31 22:07:16.340030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.339969 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" containerName="revoke-keys" Oct 31 22:07:16.340030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.340021 4294 cpu_manager.go:414] "containerMap: removing container" podUID="59404ca5-f2a4-48a4-9098-5b0c65814957" containerName="payload-processing" Oct 31 22:07:16.340030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.340027 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59404ca5-f2a4-48a4-9098-5b0c65814957" containerName="payload-processing" Oct 31 22:07:16.340439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.340199 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="59404ca5-f2a4-48a4-9098-5b0c65814957" containerName="payload-processing" Oct 31 22:07:16.340439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.340219 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="28024b1e-f5cd-48a3-b7ce-53b6f2403c29" containerName="payload-processing" Oct 31 22:07:16.340439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.340229 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bd02d6-531f-49ac-aaa6-bd6c127e686f" containerName="payload-processing" Oct 31 22:07:16.340439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.340238 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" containerName="revoke-keys" Oct 31 22:07:16.343833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.343810 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.360055 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.360029 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm"] Oct 31 22:07:16.368496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.368432 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"agc-cred-2c83ed9af069\"" Oct 31 22:07:16.368496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.368432 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"routing-overlay\"" Oct 31 22:07:16.368496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.368467 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"kube-root-ca.crt\"" Oct 31 22:07:16.368496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.368434 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"openshift-service-ca.crt\"" Oct 31 22:07:16.368828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.368550 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-plugins-e2e-extmod-b-56a6d4\"" Oct 31 22:07:16.371732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.371706 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"payload-processing-external-model-e2e--34b55c0e-dockercfg-r5jhw\"" Oct 31 22:07:16.464161 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.464123 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.464161 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.464160 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.464459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.464203 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk89r\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-kube-api-access-hk89r\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.464526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.464451 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.594231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.571423 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.594231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.571487 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.594231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.571549 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk89r\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-kube-api-access-hk89r\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.594231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.571683 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.594231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.572556 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.594231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.573871 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.594231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.578875 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.606879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.606809 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk89r\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-kube-api-access-hk89r\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.658322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.657591 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:16.966358 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:16.966331 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm"] Oct 31 22:07:16.968153 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:16.968126 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b6946d_533d_4d92_a7b4_d3e204c80175.slice/crio-b1f1dc92d1c3131a6cf27724587dd5dcdebbf6612b8cdc1de01688b78b63ae5e WatchSource:0}: Error finding container b1f1dc92d1c3131a6cf27724587dd5dcdebbf6612b8cdc1de01688b78b63ae5e: Status 404 returned error can't find the container with id b1f1dc92d1c3131a6cf27724587dd5dcdebbf6612b8cdc1de01688b78b63ae5e Oct 31 22:07:17.406769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.406737 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr"] Oct 31 22:07:17.410911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.410889 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.437922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.437871 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"payload-processing-external-model-e2e--1e97e1d3-dockercfg-nkvhj\"" Oct 31 22:07:17.438166 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.437981 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"openshift-service-ca.crt\"" Oct 31 22:07:17.438166 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.438038 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"payload-processing-external-model-plugins-e2e-extmod-a-943963\"" Oct 31 22:07:17.438166 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.438098 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"routing-overlay\"" Oct 31 22:07:17.438391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.438177 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"kube-root-ca.crt\"" Oct 31 22:07:17.438391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.438232 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"agc-cred-883ce719e9af\"" Oct 31 22:07:17.441903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.441875 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr"] Oct 31 22:07:17.494596 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.494564 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.494761 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.494671 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5rg\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-kube-api-access-6b5rg\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.494809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.494772 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.494954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.494935 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.604323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.603356 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.604323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.603447 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.604323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.603539 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5rg\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-kube-api-access-6b5rg\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.604323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.603603 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.608325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.606149 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.608325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.606829 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.611394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.611346 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.651018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.650978 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5rg\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-kube-api-access-6b5rg\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.680994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.679621 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" event={"ID":"f7b6946d-533d-4d92-a7b4-d3e204c80175","Type":"ContainerStarted","Data":"2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce"} Oct 31 22:07:17.680994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.679693 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" event={"ID":"f7b6946d-533d-4d92-a7b4-d3e204c80175","Type":"ContainerStarted","Data":"b1f1dc92d1c3131a6cf27724587dd5dcdebbf6612b8cdc1de01688b78b63ae5e"} Oct 31 22:07:17.680994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.680872 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:17.724916 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.724654 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:17.728859 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:17.728262 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" podStartSLOduration=1.728248138 podStartE2EDuration="1.728248138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:17.722063411 +0000 UTC m=+3077.847170643" watchObservedRunningTime="2026-10-31 22:07:17.728248138 +0000 UTC m=+3077.853355368" Oct 31 22:07:18.039951 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:18.039920 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr"] Oct 31 22:07:18.042170 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:18.042141 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281c0b19_d846_4e0d_9d14_6382c3ea2af6.slice/crio-a2f668c1d6699694f7b08222aaba0a76de7826dc0852cb167da4cf40ffa790ee WatchSource:0}: Error finding container a2f668c1d6699694f7b08222aaba0a76de7826dc0852cb167da4cf40ffa790ee: Status 404 returned error can't find the container with id a2f668c1d6699694f7b08222aaba0a76de7826dc0852cb167da4cf40ffa790ee Oct 31 22:07:18.688264 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:18.688225 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" event={"ID":"281c0b19-d846-4e0d-9d14-6382c3ea2af6","Type":"ContainerStarted","Data":"7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4"} Oct 31 22:07:18.688717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:18.688284 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" event={"ID":"281c0b19-d846-4e0d-9d14-6382c3ea2af6","Type":"ContainerStarted","Data":"a2f668c1d6699694f7b08222aaba0a76de7826dc0852cb167da4cf40ffa790ee"} Oct 31 22:07:18.730424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:18.730369 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" podStartSLOduration=1.730353711 podStartE2EDuration="1.730353711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:18.729378061 +0000 UTC m=+3078.854485292" watchObservedRunningTime="2026-10-31 22:07:18.730353711 +0000 UTC m=+3078.855460941" Oct 31 22:07:19.695131 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:19.695094 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:20.012346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:20.012315 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6"] Oct 31 22:07:20.012568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:20.012541 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" containerID="cri-o://3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2" gracePeriod=30 Oct 31 22:07:23.342516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.342483 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g"] Oct 31 22:07:23.354456 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.354429 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.366951 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.366928 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g"] Oct 31 22:07:23.373338 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.372704 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"agc-cred-5ee76dc49b61\"" Oct 31 22:07:23.512735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.512698 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsq4v\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-kube-api-access-wsq4v\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.512735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.512740 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.512973 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.512769 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.513016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.512967 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.614170 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.614066 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.614391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.614249 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsq4v\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-kube-api-access-wsq4v\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.614391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.614282 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.614391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.614340 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.614932 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.614908 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.615078 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.615059 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.617010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.616985 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.639187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.639157 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsq4v\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-kube-api-access-wsq4v\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868brxp2g\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.672715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.672685 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:23.975188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:23.975152 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g"] Oct 31 22:07:23.976587 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:23.976561 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f9fe2f_ea5a_451a_8b84_30b00e989c3f.slice/crio-40fda4f58643766eadb04c5966db5ad5411c0c03296d68774a51d828aea7e2ab WatchSource:0}: Error finding container 40fda4f58643766eadb04c5966db5ad5411c0c03296d68774a51d828aea7e2ab: Status 404 returned error can't find the container with id 40fda4f58643766eadb04c5966db5ad5411c0c03296d68774a51d828aea7e2ab Oct 31 22:07:24.753020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:24.752984 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" event={"ID":"74f9fe2f-ea5a-451a-8b84-30b00e989c3f","Type":"ContainerStarted","Data":"80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c"} Oct 31 22:07:24.753490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:24.753050 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" event={"ID":"74f9fe2f-ea5a-451a-8b84-30b00e989c3f","Type":"ContainerStarted","Data":"40fda4f58643766eadb04c5966db5ad5411c0c03296d68774a51d828aea7e2ab"} Oct 31 22:07:24.753490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:24.753218 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:24.802923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:24.802869 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" podStartSLOduration=1.802856436 podStartE2EDuration="1.802856436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:24.799731273 +0000 UTC m=+3084.924838507" watchObservedRunningTime="2026-10-31 22:07:24.802856436 +0000 UTC m=+3084.927963714" Oct 31 22:07:24.989214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:24.989176 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:07:24.989420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:24.989237 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:07:25.299216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.299190 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 22:07:25.440784 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.440752 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-credential-socket\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.440971 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.440795 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-podinfo\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.440971 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.440869 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-certs\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.441074 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.440974 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-envoy\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.441074 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441063 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-kube-api-access-vhpkp\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-kube-api-access-vhpkp\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.441160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441069 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-credential-socket" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:25.441160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441097 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21f55d66-e39e-46ac-8b3c-23d40268f95b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f55d66-e39e-46ac-8b3c-23d40268f95b-istiod-ca-cert\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.441264 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441159 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-token\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-token\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.441264 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441235 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-socket\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.441404 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441262 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-certs" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:25.441466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441444 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-data\") pod \"21f55d66-e39e-46ac-8b3c-23d40268f95b\" (UID: \"21f55d66-e39e-46ac-8b3c-23d40268f95b\") " Oct 31 22:07:25.441524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441467 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f55d66-e39e-46ac-8b3c-23d40268f95b-istiod-ca-cert" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:25.441647 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441621 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-socket" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:25.441799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.441770 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-data" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:25.442115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.442096 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.442115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.442116 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.442250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.442126 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.442250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.442134 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21f55d66-e39e-46ac-8b3c-23d40268f95b-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.442250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.442141 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.443365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.443339 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-envoy" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:07:25.443365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.443350 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-token" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:25.443615 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.443592 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-podinfo" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:07:25.443674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.443601 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-kube-api-access-vhpkp" pod "21f55d66-e39e-46ac-8b3c-23d40268f95b" (UID: "21f55d66-e39e-46ac-8b3c-23d40268f95b"). InnerVolumeSpecName "kube-api-access-vhpkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:25.543159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.543059 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhpkp\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-kube-api-access-vhpkp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.543159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.543098 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.543159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.543111 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.543159 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.543122 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21f55d66-e39e-46ac-8b3c-23d40268f95b-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:25.761504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.761470 4294 generic.go:356] "Generic (PLEG): container finished" podID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerID="3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2" exitCode=0 Oct 31 22:07:25.761949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.761526 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" Oct 31 22:07:25.761949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.761559 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" event={"ID":"21f55d66-e39e-46ac-8b3c-23d40268f95b","Type":"ContainerDied","Data":"3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2"} Oct 31 22:07:25.761949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.761622 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6" event={"ID":"21f55d66-e39e-46ac-8b3c-23d40268f95b","Type":"ContainerDied","Data":"d99cd5a0b3d2769aa93eb658801e18cb2927386e6185cdbfc59efa7e6983d0b4"} Oct 31 22:07:25.761949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.761651 4294 scope.go:122] "RemoveContainer" containerID="3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2" Oct 31 22:07:25.784445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.784422 4294 scope.go:122] "RemoveContainer" containerID="3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2" Oct 31 22:07:25.784824 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:07:25.784794 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2\": container with ID starting with 3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2 not found: ID does not exist" containerID="3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2" Oct 31 22:07:25.784944 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.784829 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2"} err="failed to get container status \"3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2\": rpc error: code = NotFound desc = could not find container \"3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2\": container with ID starting with 3f5bdf6b8ae26002ff5635893ca8e223c2c2ce5d1a0cfaffacb5af9c5f1860c2 not found: ID does not exist" Oct 31 22:07:25.816005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.815923 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6"] Oct 31 22:07:25.829958 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.829924 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-44405d-openshift-default-6447558b8c-4pkc6"] Oct 31 22:07:25.986314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:25.986277 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" path="/var/lib/kubelet/pods/21f55d66-e39e-46ac-8b3c-23d40268f95b/volumes" Oct 31 22:07:29.699257 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:29.699223 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:30.705976 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:30.705943 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:31.213698 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.213667 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5"] Oct 31 22:07:31.214326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.214311 4294 cpu_manager.go:414] "containerMap: removing container" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" Oct 31 22:07:31.214326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.214327 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" Oct 31 22:07:31.214447 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.214438 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="21f55d66-e39e-46ac-8b3c-23d40268f95b" containerName="istio-proxy" Oct 31 22:07:31.219474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.219448 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.235149 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.235118 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:07:31.235287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.235241 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:07:31.235914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.235888 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5"] Oct 31 22:07:31.330564 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.330527 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.336448 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.336412 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.336915 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.336893 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p62v\" (UniqueName: \"kubernetes.io/projected/411b519c-ad33-4941-ba57-4e8fe32c8614-kube-api-access-7p62v\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.465383 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.438436 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p62v\" (UniqueName: \"kubernetes.io/projected/411b519c-ad33-4941-ba57-4e8fe32c8614-kube-api-access-7p62v\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.474335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.465772 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.495572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.495529 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.497022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.496992 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.571330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.571247 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.591794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.591732 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p62v\" (UniqueName: \"kubernetes.io/projected/411b519c-ad33-4941-ba57-4e8fe32c8614-kube-api-access-7p62v\") pod \"maas-api-revoke-sub-4ee3803f-k2wc5\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:31.639533 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:07:31.639497 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:07:31.843383 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:31.842696 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:32.214375 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:32.214346 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5"] Oct 31 22:07:32.214544 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:32.214517 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411b519c_ad33_4941_ba57_4e8fe32c8614.slice/crio-7c7a34d769c52f7bffbe10d17163d2182cfd2d4677a1e64ea1918ea424060cb3 WatchSource:0}: Error finding container 7c7a34d769c52f7bffbe10d17163d2182cfd2d4677a1e64ea1918ea424060cb3: Status 404 returned error can't find the container with id 7c7a34d769c52f7bffbe10d17163d2182cfd2d4677a1e64ea1918ea424060cb3 Oct 31 22:07:32.851343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:32.851292 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" event={"ID":"411b519c-ad33-4941-ba57-4e8fe32c8614","Type":"ContainerStarted","Data":"a32c4bbe856b4bad8197565a5869382476740110bd1a94cc30695396e2ca2030"} Oct 31 22:07:32.851790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:32.851398 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" event={"ID":"411b519c-ad33-4941-ba57-4e8fe32c8614","Type":"ContainerStarted","Data":"7c7a34d769c52f7bffbe10d17163d2182cfd2d4677a1e64ea1918ea424060cb3"} Oct 31 22:07:32.893627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:32.893579 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" podStartSLOduration=1.893565997 podStartE2EDuration="1.893565997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:32.890650256 +0000 UTC m=+3093.015757496" watchObservedRunningTime="2026-10-31 22:07:32.893565997 +0000 UTC m=+3093.018673227" Oct 31 22:07:33.189639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.189557 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl"] Oct 31 22:07:33.192973 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.192951 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.210291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.210265 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-639a39\"/\"openshift-service-ca.crt\"" Oct 31 22:07:33.211760 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.210432 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-639a39\"/\"kube-root-ca.crt\"" Oct 31 22:07:33.211760 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.210550 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-639a39\"/\"default-dockercfg-mmmf6\"" Oct 31 22:07:33.211760 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.210987 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-639a39\"/\"e2e-usage-sim-bd9aba-cert\"" Oct 31 22:07:33.212623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.212601 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl"] Oct 31 22:07:33.265232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.265200 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbbf\" (UniqueName: \"kubernetes.io/projected/a1e93389-0027-4e47-adb2-176eef3e5880-kube-api-access-jtbbf\") pod \"e2e-usage-sim-bd9aba-86b8fbc958-lmwbl\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.265431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.265402 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1e93389-0027-4e47-adb2-176eef3e5880-tls\") pod \"e2e-usage-sim-bd9aba-86b8fbc958-lmwbl\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.367397 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.367369 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbbf\" (UniqueName: \"kubernetes.io/projected/a1e93389-0027-4e47-adb2-176eef3e5880-kube-api-access-jtbbf\") pod \"e2e-usage-sim-bd9aba-86b8fbc958-lmwbl\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.367537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.367482 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1e93389-0027-4e47-adb2-176eef3e5880-tls\") pod \"e2e-usage-sim-bd9aba-86b8fbc958-lmwbl\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.370680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.370652 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1e93389-0027-4e47-adb2-176eef3e5880-tls\") pod \"e2e-usage-sim-bd9aba-86b8fbc958-lmwbl\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.391136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.391107 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbbf\" (UniqueName: \"kubernetes.io/projected/a1e93389-0027-4e47-adb2-176eef3e5880-kube-api-access-jtbbf\") pod \"e2e-usage-sim-bd9aba-86b8fbc958-lmwbl\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.515183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.515151 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:33.813759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.813727 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl"] Oct 31 22:07:33.815013 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:33.814987 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1e93389_0027_4e47_adb2_176eef3e5880.slice/crio-3b7f232e3f4627c3377583411c12eeeff7ebd5eb8807743eea10fdab95361034 WatchSource:0}: Error finding container 3b7f232e3f4627c3377583411c12eeeff7ebd5eb8807743eea10fdab95361034: Status 404 returned error can't find the container with id 3b7f232e3f4627c3377583411c12eeeff7ebd5eb8807743eea10fdab95361034 Oct 31 22:07:33.863412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.863377 4294 generic.go:356] "Generic (PLEG): container finished" podID="411b519c-ad33-4941-ba57-4e8fe32c8614" containerID="a32c4bbe856b4bad8197565a5869382476740110bd1a94cc30695396e2ca2030" exitCode=0 Oct 31 22:07:33.863898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.863858 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" event={"ID":"411b519c-ad33-4941-ba57-4e8fe32c8614","Type":"ContainerDied","Data":"a32c4bbe856b4bad8197565a5869382476740110bd1a94cc30695396e2ca2030"} Oct 31 22:07:33.865415 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:33.865387 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" event={"ID":"a1e93389-0027-4e47-adb2-176eef3e5880","Type":"ContainerStarted","Data":"3b7f232e3f4627c3377583411c12eeeff7ebd5eb8807743eea10fdab95361034"} Oct 31 22:07:34.874152 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:34.874112 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" event={"ID":"a1e93389-0027-4e47-adb2-176eef3e5880","Type":"ContainerStarted","Data":"fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787"} Oct 31 22:07:34.874664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:34.874285 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:34.879843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:34.879821 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:07:34.915563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:34.915512 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" podStartSLOduration=1.9154921329999999 podStartE2EDuration="1.915492133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:34.913799385 +0000 UTC m=+3095.038906616" watchObservedRunningTime="2026-10-31 22:07:34.915492133 +0000 UTC m=+3095.040599369" Oct 31 22:07:35.034239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.034215 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:35.193278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.193200 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-openshift-ca\") pod \"411b519c-ad33-4941-ba57-4e8fe32c8614\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " Oct 31 22:07:35.193470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.193377 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411b519c-ad33-4941-ba57-4e8fe32c8614-kube-api-access-7p62v\" (UniqueName: \"kubernetes.io/projected/411b519c-ad33-4941-ba57-4e8fe32c8614-kube-api-access-7p62v\") pod \"411b519c-ad33-4941-ba57-4e8fe32c8614\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " Oct 31 22:07:35.193543 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.193466 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-xks-ca\") pod \"411b519c-ad33-4941-ba57-4e8fe32c8614\" (UID: \"411b519c-ad33-4941-ba57-4e8fe32c8614\") " Oct 31 22:07:35.193593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.193574 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-openshift-ca" pod "411b519c-ad33-4941-ba57-4e8fe32c8614" (UID: "411b519c-ad33-4941-ba57-4e8fe32c8614"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:35.194135 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.194110 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:35.195595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.195572 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411b519c-ad33-4941-ba57-4e8fe32c8614-kube-api-access-7p62v" pod "411b519c-ad33-4941-ba57-4e8fe32c8614" (UID: "411b519c-ad33-4941-ba57-4e8fe32c8614"). InnerVolumeSpecName "kube-api-access-7p62v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:35.195706 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.195613 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-xks-ca" pod "411b519c-ad33-4941-ba57-4e8fe32c8614" (UID: "411b519c-ad33-4941-ba57-4e8fe32c8614"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:07:35.295235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.295208 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411b519c-ad33-4941-ba57-4e8fe32c8614-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:35.295235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.295233 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p62v\" (UniqueName: \"kubernetes.io/projected/411b519c-ad33-4941-ba57-4e8fe32c8614-kube-api-access-7p62v\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:35.764975 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.764948 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:35.841986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.841952 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr"] Oct 31 22:07:35.883684 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.883653 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" Oct 31 22:07:35.884497 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.883832 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5" event={"ID":"411b519c-ad33-4941-ba57-4e8fe32c8614","Type":"ContainerDied","Data":"7c7a34d769c52f7bffbe10d17163d2182cfd2d4677a1e64ea1918ea424060cb3"} Oct 31 22:07:35.884612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:35.884516 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7a34d769c52f7bffbe10d17163d2182cfd2d4677a1e64ea1918ea424060cb3" Oct 31 22:07:36.873086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:36.873046 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm"] Oct 31 22:07:40.664226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.664195 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8"] Oct 31 22:07:40.664915 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.664899 4294 cpu_manager.go:414] "containerMap: removing container" podUID="411b519c-ad33-4941-ba57-4e8fe32c8614" containerName="revoke-keys" Oct 31 22:07:40.664972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.664918 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411b519c-ad33-4941-ba57-4e8fe32c8614" containerName="revoke-keys" Oct 31 22:07:40.665060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.665050 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="411b519c-ad33-4941-ba57-4e8fe32c8614" containerName="revoke-keys" Oct 31 22:07:40.668223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.668202 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.688063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.688034 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8"] Oct 31 22:07:40.694753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.694728 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"agc-cred-ddfff8df45ca\"" Oct 31 22:07:40.695731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.695713 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-56a6d4\"/\"agc-cred-4379f1a3a622\"" Oct 31 22:07:40.781757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.781725 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.781889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.781794 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.781952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.781929 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.782005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.781969 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwln\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-kube-api-access-4fwln\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.842405 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.842356 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" podUID="281c0b19-d846-4e0d-9d14-6382c3ea2af6" containerName="payload-processing" containerID="cri-o://7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4" gracePeriod=25 Oct 31 22:07:40.883678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.883652 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.883775 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.883724 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.883836 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.883820 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.883872 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.883861 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwln\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-kube-api-access-4fwln\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.884473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.884446 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.884630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.884527 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.886379 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.886347 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.907285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.907265 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwln\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-kube-api-access-4fwln\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:40.985593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:40.985565 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:41.140200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.140177 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:41.291741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.291707 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-plugins-config-volume\") pod \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " Oct 31 22:07:41.292107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.291789 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-kube-api-access-6b5rg\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-kube-api-access-6b5rg\") pod \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " Oct 31 22:07:41.292107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.291995 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-routing-overlay-volume\") pod \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " Oct 31 22:07:41.292784 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.292526 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-plugins-config-volume" pod "281c0b19-d846-4e0d-9d14-6382c3ea2af6" (UID: "281c0b19-d846-4e0d-9d14-6382c3ea2af6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:41.293016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.292992 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-provider-credentials\") pod \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\" (UID: \"281c0b19-d846-4e0d-9d14-6382c3ea2af6\") " Oct 31 22:07:41.293115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.293039 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-routing-overlay-volume" pod "281c0b19-d846-4e0d-9d14-6382c3ea2af6" (UID: "281c0b19-d846-4e0d-9d14-6382c3ea2af6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:41.293651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.293628 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:41.293765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.293655 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/281c0b19-d846-4e0d-9d14-6382c3ea2af6-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:41.295210 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.295189 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-kube-api-access-6b5rg" pod "281c0b19-d846-4e0d-9d14-6382c3ea2af6" (UID: "281c0b19-d846-4e0d-9d14-6382c3ea2af6"). InnerVolumeSpecName "kube-api-access-6b5rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:41.295576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.295553 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-provider-credentials" pod "281c0b19-d846-4e0d-9d14-6382c3ea2af6" (UID: "281c0b19-d846-4e0d-9d14-6382c3ea2af6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:41.304974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.304952 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8"] Oct 31 22:07:41.305493 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:41.305472 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafda436_26c7_4192_898b_bba67a024214.slice/crio-a3b716e66dcbc279518f05839afb711df78491dce677c4ad227c90150fcfd6e4 WatchSource:0}: Error finding container a3b716e66dcbc279518f05839afb711df78491dce677c4ad227c90150fcfd6e4: Status 404 returned error can't find the container with id a3b716e66dcbc279518f05839afb711df78491dce677c4ad227c90150fcfd6e4 Oct 31 22:07:41.395270 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.395236 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:41.395270 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.395261 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b5rg\" (UniqueName: \"kubernetes.io/projected/281c0b19-d846-4e0d-9d14-6382c3ea2af6-kube-api-access-6b5rg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:41.484764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.484737 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6"] Oct 31 22:07:41.485410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.485396 4294 cpu_manager.go:414] "containerMap: removing container" podUID="281c0b19-d846-4e0d-9d14-6382c3ea2af6" containerName="payload-processing" Oct 31 22:07:41.485465 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.485411 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281c0b19-d846-4e0d-9d14-6382c3ea2af6" containerName="payload-processing" Oct 31 22:07:41.485561 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.485552 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="281c0b19-d846-4e0d-9d14-6382c3ea2af6" containerName="payload-processing" Oct 31 22:07:41.489099 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.489078 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.504978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.504955 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-943963\"/\"agc-cred-24e1cf1a8064\"" Oct 31 22:07:41.509229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.508974 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6"] Oct 31 22:07:41.597438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.597365 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jhbl\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-kube-api-access-9jhbl\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.597438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.597423 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.597675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.597656 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.597776 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.597755 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.699375 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.699349 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.699805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.699483 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jhbl\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-kube-api-access-9jhbl\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.699805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.699525 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.699805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.699672 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.700226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.700206 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.700293 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.700260 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.702113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.702093 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.723100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.723068 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jhbl\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-kube-api-access-9jhbl\") pod \"payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.803811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.803788 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:41.884361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.874361 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" podUID="f7b6946d-533d-4d92-a7b4-d3e204c80175" containerName="payload-processing" containerID="cri-o://2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce" gracePeriod=25 Oct 31 22:07:41.952972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.952939 4294 generic.go:356] "Generic (PLEG): container finished" podID="281c0b19-d846-4e0d-9d14-6382c3ea2af6" containerID="7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4" exitCode=0 Oct 31 22:07:41.953130 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.953029 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" event={"ID":"281c0b19-d846-4e0d-9d14-6382c3ea2af6","Type":"ContainerDied","Data":"7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4"} Oct 31 22:07:41.953130 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.953067 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" event={"ID":"281c0b19-d846-4e0d-9d14-6382c3ea2af6","Type":"ContainerDied","Data":"a2f668c1d6699694f7b08222aaba0a76de7826dc0852cb167da4cf40ffa790ee"} Oct 31 22:07:41.953130 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.953088 4294 scope.go:122] "RemoveContainer" containerID="7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4" Oct 31 22:07:41.953337 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.953203 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr" Oct 31 22:07:41.957683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.957630 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" event={"ID":"eafda436-26c7-4192-898b-bba67a024214","Type":"ContainerStarted","Data":"c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0"} Oct 31 22:07:41.957805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.957684 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" event={"ID":"eafda436-26c7-4192-898b-bba67a024214","Type":"ContainerStarted","Data":"a3b716e66dcbc279518f05839afb711df78491dce677c4ad227c90150fcfd6e4"} Oct 31 22:07:41.957805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.957715 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:41.988538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.988503 4294 scope.go:122] "RemoveContainer" containerID="7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4" Oct 31 22:07:41.988861 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:07:41.988838 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4\": container with ID starting with 7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4 not found: ID does not exist" containerID="7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4" Oct 31 22:07:41.988922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:41.988874 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4"} err="failed to get container status \"7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4\": rpc error: code = NotFound desc = could not find container \"7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4\": container with ID starting with 7fbc536acd6f7479373b75008b05ff79c99e6508f496e48a7d7df7c16608daf4 not found: ID does not exist" Oct 31 22:07:42.013366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.013323 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" podStartSLOduration=2.013293567 podStartE2EDuration="2.013293567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:42.012171798 +0000 UTC m=+3102.137279031" watchObservedRunningTime="2026-10-31 22:07:42.013293567 +0000 UTC m=+3102.138400798" Oct 31 22:07:42.058980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.058943 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr"] Oct 31 22:07:42.066923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.062759 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb62htkr"] Oct 31 22:07:42.117282 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.117251 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6"] Oct 31 22:07:42.119815 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:42.119789 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6c0b2b_4ceb_404d_900f_6b351268964c.slice/crio-7596d8367b6dfbc398ee4acf704cef031b8170146efcf90eeefe9ca06ef8d92f WatchSource:0}: Error finding container 7596d8367b6dfbc398ee4acf704cef031b8170146efcf90eeefe9ca06ef8d92f: Status 404 returned error can't find the container with id 7596d8367b6dfbc398ee4acf704cef031b8170146efcf90eeefe9ca06ef8d92f Oct 31 22:07:42.251831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.251806 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:42.417026 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.416932 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-plugins-config-volume\") pod \"f7b6946d-533d-4d92-a7b4-d3e204c80175\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " Oct 31 22:07:42.417191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.417176 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-routing-overlay-volume\") pod \"f7b6946d-533d-4d92-a7b4-d3e204c80175\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " Oct 31 22:07:42.417260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.417206 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-provider-credentials\") pod \"f7b6946d-533d-4d92-a7b4-d3e204c80175\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " Oct 31 22:07:42.417371 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.417351 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-kube-api-access-hk89r\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-kube-api-access-hk89r\") pod \"f7b6946d-533d-4d92-a7b4-d3e204c80175\" (UID: \"f7b6946d-533d-4d92-a7b4-d3e204c80175\") " Oct 31 22:07:42.417444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.417381 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-plugins-config-volume" pod "f7b6946d-533d-4d92-a7b4-d3e204c80175" (UID: "f7b6946d-533d-4d92-a7b4-d3e204c80175"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:42.417591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.417570 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-routing-overlay-volume" pod "f7b6946d-533d-4d92-a7b4-d3e204c80175" (UID: "f7b6946d-533d-4d92-a7b4-d3e204c80175"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:42.418028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.418005 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:42.418028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.418030 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7b6946d-533d-4d92-a7b4-d3e204c80175-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:42.419590 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.419568 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-provider-credentials" pod "f7b6946d-533d-4d92-a7b4-d3e204c80175" (UID: "f7b6946d-533d-4d92-a7b4-d3e204c80175"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:42.419689 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.419601 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-kube-api-access-hk89r" pod "f7b6946d-533d-4d92-a7b4-d3e204c80175" (UID: "f7b6946d-533d-4d92-a7b4-d3e204c80175"). InnerVolumeSpecName "kube-api-access-hk89r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:42.519163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.519128 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk89r\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-kube-api-access-hk89r\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:42.519163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.519156 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f7b6946d-533d-4d92-a7b4-d3e204c80175-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:42.967415 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.967375 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" event={"ID":"bf6c0b2b-4ceb-404d-900f-6b351268964c","Type":"ContainerStarted","Data":"17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313"} Oct 31 22:07:42.967901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.967440 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" event={"ID":"bf6c0b2b-4ceb-404d-900f-6b351268964c","Type":"ContainerStarted","Data":"7596d8367b6dfbc398ee4acf704cef031b8170146efcf90eeefe9ca06ef8d92f"} Oct 31 22:07:42.967901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.967588 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:42.969076 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.969050 4294 generic.go:356] "Generic (PLEG): container finished" podID="f7b6946d-533d-4d92-a7b4-d3e204c80175" containerID="2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce" exitCode=0 Oct 31 22:07:42.969229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.969083 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" event={"ID":"f7b6946d-533d-4d92-a7b4-d3e204c80175","Type":"ContainerDied","Data":"2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce"} Oct 31 22:07:42.969229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.969108 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" Oct 31 22:07:42.969229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.969129 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm" event={"ID":"f7b6946d-533d-4d92-a7b4-d3e204c80175","Type":"ContainerDied","Data":"b1f1dc92d1c3131a6cf27724587dd5dcdebbf6612b8cdc1de01688b78b63ae5e"} Oct 31 22:07:42.969229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.969160 4294 scope.go:122] "RemoveContainer" containerID="2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce" Oct 31 22:07:42.979900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.979884 4294 scope.go:122] "RemoveContainer" containerID="2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce" Oct 31 22:07:42.980171 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:07:42.980154 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce\": container with ID starting with 2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce not found: ID does not exist" containerID="2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce" Oct 31 22:07:42.980208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:42.980180 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce"} err="failed to get container status \"2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce\": rpc error: code = NotFound desc = could not find container \"2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce\": container with ID starting with 2740ac00e882a7f080f74bfd0cc048180147760384035a5f1accc30a08ea51ce not found: ID does not exist" Oct 31 22:07:43.010693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:43.010650 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" podStartSLOduration=2.010636336 podStartE2EDuration="2.010636336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:43.00837959 +0000 UTC m=+3103.133486831" watchObservedRunningTime="2026-10-31 22:07:43.010636336 +0000 UTC m=+3103.135743566" Oct 31 22:07:43.046984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:43.046951 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm"] Oct 31 22:07:43.050558 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:43.050534 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4cmprwm"] Oct 31 22:07:43.985139 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:43.985095 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281c0b19-d846-4e0d-9d14-6382c3ea2af6" path="/var/lib/kubelet/pods/281c0b19-d846-4e0d-9d14-6382c3ea2af6/volumes" Oct 31 22:07:43.985783 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:43.985758 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b6946d-533d-4d92-a7b4-d3e204c80175" path="/var/lib/kubelet/pods/f7b6946d-533d-4d92-a7b4-d3e204c80175/volumes" Oct 31 22:07:44.225455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.225412 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-54fc546546-rdlkm"] Oct 31 22:07:44.226087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.226071 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f7b6946d-533d-4d92-a7b4-d3e204c80175" containerName="payload-processing" Oct 31 22:07:44.226087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.226086 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b6946d-533d-4d92-a7b4-d3e204c80175" containerName="payload-processing" Oct 31 22:07:44.226267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.226180 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b6946d-533d-4d92-a7b4-d3e204c80175" containerName="payload-processing" Oct 31 22:07:44.231152 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.231130 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.244218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.244189 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-plugins-config-volume\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.244400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.244292 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmjb5\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-kube-api-access-kmjb5\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.244538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.244520 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-provider-credentials\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.244668 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.244648 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-routing-overlay-volume\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.251135 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.251108 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fc546546-rdlkm"] Oct 31 22:07:44.353365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.346855 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-provider-credentials\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.353365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.347038 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-routing-overlay-volume\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.353365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.347116 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-plugins-config-volume\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.353365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.347198 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmjb5\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-kube-api-access-kmjb5\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.392384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.370231 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-routing-overlay-volume\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.392384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.386178 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-plugins-config-volume\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.417323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.400322 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmjb5\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-kube-api-access-kmjb5\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.417323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.400911 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-provider-credentials\") pod \"payload-processing-external-model-54fc546546-rdlkm\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.545364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.545258 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:44.846473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.846441 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fc546546-rdlkm"] Oct 31 22:07:44.847470 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:44.847443 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a35efad_1d5e_4cc8_ac0f_5e65dcb15606.slice/crio-ca9283d98673e8f2e9ca3d81e2cf6642ff6da5617ad1c0d84c9f202935973741 WatchSource:0}: Error finding container ca9283d98673e8f2e9ca3d81e2cf6642ff6da5617ad1c0d84c9f202935973741: Status 404 returned error can't find the container with id ca9283d98673e8f2e9ca3d81e2cf6642ff6da5617ad1c0d84c9f202935973741 Oct 31 22:07:44.992070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.991668 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" event={"ID":"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606","Type":"ContainerStarted","Data":"93207883830d15726692b4c76bd4d3ee421ffd83a8c5fda8227c59db37e4e8ec"} Oct 31 22:07:44.992070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.991732 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" event={"ID":"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606","Type":"ContainerStarted","Data":"ca9283d98673e8f2e9ca3d81e2cf6642ff6da5617ad1c0d84c9f202935973741"} Oct 31 22:07:44.992533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:44.992113 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:45.034121 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:45.034071 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" podStartSLOduration=1.0340583 podStartE2EDuration="1.0340583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:45.033383755 +0000 UTC m=+3105.158490987" watchObservedRunningTime="2026-10-31 22:07:45.0340583 +0000 UTC m=+3105.159165551" Oct 31 22:07:52.197592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.197549 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt"] Oct 31 22:07:52.202719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.202695 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.220739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.220714 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt"] Oct 31 22:07:52.234147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.227265 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-710927\"" Oct 31 22:07:52.301768 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.301730 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f451d27-023c-467d-b582-d8373e0da7fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.301952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.301782 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4r2f\" (UniqueName: \"kubernetes.io/projected/9f451d27-023c-467d-b582-d8373e0da7fe-kube-api-access-m4r2f\") pod \"payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.408344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.405489 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f451d27-023c-467d-b582-d8373e0da7fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.408344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.405557 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4r2f\" (UniqueName: \"kubernetes.io/projected/9f451d27-023c-467d-b582-d8373e0da7fe-kube-api-access-m4r2f\") pod \"payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.408344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.406971 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f451d27-023c-467d-b582-d8373e0da7fe-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.478060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.477962 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4r2f\" (UniqueName: \"kubernetes.io/projected/9f451d27-023c-467d-b582-d8373e0da7fe-kube-api-access-m4r2f\") pod \"payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.522742 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.522708 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:52.823360 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.823330 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt"] Oct 31 22:07:52.824098 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:07:52.824070 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f451d27_023c_467d_b582_d8373e0da7fe.slice/crio-43ba612ba40d6a9834d34dfed58f7a13792394955a84ac1b00a8f0ba7ffec792 WatchSource:0}: Error finding container 43ba612ba40d6a9834d34dfed58f7a13792394955a84ac1b00a8f0ba7ffec792: Status 404 returned error can't find the container with id 43ba612ba40d6a9834d34dfed58f7a13792394955a84ac1b00a8f0ba7ffec792 Oct 31 22:07:52.974048 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:52.974019 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:07:53.069590 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:53.069551 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" event={"ID":"9f451d27-023c-467d-b582-d8373e0da7fe","Type":"ContainerStarted","Data":"e0af8a652c62fd737a999a3bb8b41748f9360e2895fa12fc044eb99e3391c241"} Oct 31 22:07:53.069796 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:53.069615 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" event={"ID":"9f451d27-023c-467d-b582-d8373e0da7fe","Type":"ContainerStarted","Data":"43ba612ba40d6a9834d34dfed58f7a13792394955a84ac1b00a8f0ba7ffec792"} Oct 31 22:07:53.069796 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:53.069699 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:07:53.114912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:53.114819 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" podStartSLOduration=1.11480337 podStartE2EDuration="1.11480337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:07:53.111903746 +0000 UTC m=+3113.237010989" watchObservedRunningTime="2026-10-31 22:07:53.11480337 +0000 UTC m=+3113.239910615" Oct 31 22:07:53.985552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:53.985524 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:07:54.065395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:54.065363 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g"] Oct 31 22:07:56.007481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:56.007454 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:07:56.084054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:56.084016 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq"] Oct 31 22:07:59.034874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.034841 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875"] Oct 31 22:07:59.046638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.046610 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dae194f7-qw875"] Oct 31 22:07:59.066164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.066127 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" podUID="74f9fe2f-ea5a-451a-8b84-30b00e989c3f" containerName="payload-processing" containerID="cri-o://80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c" gracePeriod=25 Oct 31 22:07:59.334723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.334701 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:07:59.506379 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.506340 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-kube-api-access-wsq4v\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-kube-api-access-wsq4v\") pod \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " Oct 31 22:07:59.506559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.506433 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-provider-credentials\") pod \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " Oct 31 22:07:59.506748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.506728 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-routing-overlay-volume\") pod \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " Oct 31 22:07:59.506813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.506763 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-plugins-config-volume\") pod \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\" (UID: \"74f9fe2f-ea5a-451a-8b84-30b00e989c3f\") " Oct 31 22:07:59.507120 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.507090 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-routing-overlay-volume" pod "74f9fe2f-ea5a-451a-8b84-30b00e989c3f" (UID: "74f9fe2f-ea5a-451a-8b84-30b00e989c3f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:59.507245 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.507174 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-plugins-config-volume" pod "74f9fe2f-ea5a-451a-8b84-30b00e989c3f" (UID: "74f9fe2f-ea5a-451a-8b84-30b00e989c3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:07:59.507384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.507370 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:59.507431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.507387 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:59.508684 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.508662 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-kube-api-access-wsq4v" pod "74f9fe2f-ea5a-451a-8b84-30b00e989c3f" (UID: "74f9fe2f-ea5a-451a-8b84-30b00e989c3f"). InnerVolumeSpecName "kube-api-access-wsq4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:59.508762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.508697 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-provider-credentials" pod "74f9fe2f-ea5a-451a-8b84-30b00e989c3f" (UID: "74f9fe2f-ea5a-451a-8b84-30b00e989c3f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:07:59.608621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.608539 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsq4v\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-kube-api-access-wsq4v\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:59.608621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.608571 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/74f9fe2f-ea5a-451a-8b84-30b00e989c3f-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:07:59.986572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:07:59.986537 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6db49cf9-781a-4b25-8dae-56d00c4352f7" path="/var/lib/kubelet/pods/6db49cf9-781a-4b25-8dae-56d00c4352f7/volumes" Oct 31 22:08:00.128074 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.128040 4294 generic.go:356] "Generic (PLEG): container finished" podID="74f9fe2f-ea5a-451a-8b84-30b00e989c3f" containerID="80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c" exitCode=0 Oct 31 22:08:00.128598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.128093 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" Oct 31 22:08:00.128598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.128115 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" event={"ID":"74f9fe2f-ea5a-451a-8b84-30b00e989c3f","Type":"ContainerDied","Data":"80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c"} Oct 31 22:08:00.128598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.128172 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g" event={"ID":"74f9fe2f-ea5a-451a-8b84-30b00e989c3f","Type":"ContainerDied","Data":"40fda4f58643766eadb04c5966db5ad5411c0c03296d68774a51d828aea7e2ab"} Oct 31 22:08:00.128598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.128206 4294 scope.go:122] "RemoveContainer" containerID="80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c" Oct 31 22:08:00.138546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.138520 4294 scope.go:122] "RemoveContainer" containerID="80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c" Oct 31 22:08:00.138857 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:08:00.138828 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c\": container with ID starting with 80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c not found: ID does not exist" containerID="80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c" Oct 31 22:08:00.138961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.138862 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c"} err="failed to get container status \"80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c\": rpc error: code = NotFound desc = could not find container \"80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c\": container with ID starting with 80664149f2b423c0962a4e348d0222f3322ae61f76da13c05a92a387b6ccf99c not found: ID does not exist" Oct 31 22:08:00.181498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.181463 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g"] Oct 31 22:08:00.193125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:00.193096 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868brxp2g"] Oct 31 22:08:01.085386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.085346 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" podUID="6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" containerName="payload-processing" containerID="cri-o://b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245" gracePeriod=25 Oct 31 22:08:01.336137 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:08:01.336014 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:08:01.379924 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.379898 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:08:01.533355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.533322 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-provider-credentials\") pod \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " Oct 31 22:08:01.533542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.533380 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-plugins-config-volume\") pod \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " Oct 31 22:08:01.533542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.533442 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-kube-api-access-87nzh\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-kube-api-access-87nzh\") pod \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " Oct 31 22:08:01.533648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.533630 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-routing-overlay-volume\") pod \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\" (UID: \"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a\") " Oct 31 22:08:01.533838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.533811 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-plugins-config-volume" pod "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" (UID: "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:01.534038 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.534008 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-routing-overlay-volume" pod "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" (UID: "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:01.534354 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.534335 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:01.534354 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.534353 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:01.535708 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.535682 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-kube-api-access-87nzh" pod "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" (UID: "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a"). InnerVolumeSpecName "kube-api-access-87nzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:01.535790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.535718 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-provider-credentials" pod "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" (UID: "6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:01.636041 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.635955 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:01.636041 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.635986 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87nzh\" (UniqueName: \"kubernetes.io/projected/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a-kube-api-access-87nzh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:01.985641 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:01.985613 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f9fe2f-ea5a-451a-8b84-30b00e989c3f" path="/var/lib/kubelet/pods/74f9fe2f-ea5a-451a-8b84-30b00e989c3f/volumes" Oct 31 22:08:02.146353 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.146317 4294 generic.go:356] "Generic (PLEG): container finished" podID="6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" containerID="b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245" exitCode=0 Oct 31 22:08:02.146538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.146390 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" Oct 31 22:08:02.146538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.146389 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" event={"ID":"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a","Type":"ContainerDied","Data":"b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245"} Oct 31 22:08:02.146538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.146451 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq" event={"ID":"6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a","Type":"ContainerDied","Data":"e44918c9b8c04bb421e94cda0d73c696239f9421b14f99a9d1dabe69317dbda9"} Oct 31 22:08:02.146538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.146474 4294 scope.go:122] "RemoveContainer" containerID="b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245" Oct 31 22:08:02.157226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.157201 4294 scope.go:122] "RemoveContainer" containerID="b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245" Oct 31 22:08:02.157557 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:08:02.157532 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245\": container with ID starting with b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245 not found: ID does not exist" containerID="b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245" Oct 31 22:08:02.157634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.157565 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245"} err="failed to get container status \"b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245\": rpc error: code = NotFound desc = could not find container \"b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245\": container with ID starting with b90d1c05c377b13f7b86357c24489cdb9eb620b92d2dbe2b199a04a1bd946245 not found: ID does not exist" Oct 31 22:08:02.199745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.199722 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq"] Oct 31 22:08:02.211453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.211432 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-58cdbd54fd-wbdtq"] Oct 31 22:08:02.895377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.895347 4294 scope.go:122] "RemoveContainer" containerID="e8a0017aa956a3b42dddeadef349785de71e5f9d58c27c29e878aedf3add8cf9" Oct 31 22:08:02.905069 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.905041 4294 scope.go:122] "RemoveContainer" containerID="d81cf0c018f7920d572c490161fdfb87028f5579b9ac488db45536546b114695" Oct 31 22:08:02.914089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.914072 4294 scope.go:122] "RemoveContainer" containerID="b61a635795a66fd5eb692e3c5e23ae9e4422bcb383a3baefa108363be58dec76" Oct 31 22:08:02.922359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.922341 4294 scope.go:122] "RemoveContainer" containerID="b19358c898d51a52e09d826b75c539a6aa2be5301b6ca033e7ac55589f2344da" Oct 31 22:08:02.930320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.930277 4294 scope.go:122] "RemoveContainer" containerID="a3b577c5c40d8aee838cf803131ec08f06e2aba093d3f8a058f482448bcb012e" Oct 31 22:08:02.938585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:02.938567 4294 scope.go:122] "RemoveContainer" containerID="c86b0f868e0a196fd238f4cbfba4a6d8fa0f23600170b034ed5e80c4d88a5416" Oct 31 22:08:04.005112 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:04.005078 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" path="/var/lib/kubelet/pods/6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a/volumes" Oct 31 22:08:04.082662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:04.082635 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:08:06.754436 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:06.754394 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-5drlf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:08:06.755044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:06.755007 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.255:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:08:06.843431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:06.843394 4294 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-kf6vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:08:06.843686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:06.843654 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:08:10.322446 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.322411 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs"] Oct 31 22:08:10.323367 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.323350 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" containerName="payload-processing" Oct 31 22:08:10.323428 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.323369 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" containerName="payload-processing" Oct 31 22:08:10.323466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.323432 4294 cpu_manager.go:414] "containerMap: removing container" podUID="74f9fe2f-ea5a-451a-8b84-30b00e989c3f" containerName="payload-processing" Oct 31 22:08:10.323466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.323440 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f9fe2f-ea5a-451a-8b84-30b00e989c3f" containerName="payload-processing" Oct 31 22:08:10.323589 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.323578 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe7893c-9dc9-43ef-8a4f-b03c7e1a013a" containerName="payload-processing" Oct 31 22:08:10.323624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.323604 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f9fe2f-ea5a-451a-8b84-30b00e989c3f" containerName="payload-processing" Oct 31 22:08:10.329402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.329384 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.478949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.478845 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs"] Oct 31 22:08:10.480697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.480664 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.481784 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.481758 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.487632 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.487603 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.487877 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.487857 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm4m9\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-kube-api-access-cm4m9\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492201 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"agc-cred-c0989a1222cc\"" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492344 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-e2e--956cda1a-dockercfg-mccg7\"" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492434 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"routing-overlay\"" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492487 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"payload-processing-external-model-plugins-e2e-extmod-a-696f9a\"" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492558 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"kube-root-ca.crt\"" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492621 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"openshift-service-ca.crt\"" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492693 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"agc-cred-ca74aeda362a\"" Oct 31 22:08:10.493662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.492741 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-696f9a\"/\"agc-cred-0d8dc58753dc\"" Oct 31 22:08:10.589312 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.589275 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.589501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.589427 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.589501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.589450 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm4m9\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-kube-api-access-cm4m9\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.589619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.589580 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.590033 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.590003 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.590182 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.590159 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.592315 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.592277 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.613229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.613200 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm4m9\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-kube-api-access-cm4m9\") pod \"payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.643526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.643501 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:10.956948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:10.956916 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs"] Oct 31 22:08:10.957201 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:08:10.957177 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3510a35_28bc_4079_b5ea_afa29eb58d08.slice/crio-3165c841189aa6e0744686ee6ee1261439cc9a472ce7ea0bf03e03473af77f46 WatchSource:0}: Error finding container 3165c841189aa6e0744686ee6ee1261439cc9a472ce7ea0bf03e03473af77f46: Status 404 returned error can't find the container with id 3165c841189aa6e0744686ee6ee1261439cc9a472ce7ea0bf03e03473af77f46 Oct 31 22:08:11.322168 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:11.322116 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" event={"ID":"a3510a35-28bc-4079-b5ea-afa29eb58d08","Type":"ContainerStarted","Data":"a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444"} Oct 31 22:08:11.322372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:11.322226 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:11.322372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:11.322250 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" event={"ID":"a3510a35-28bc-4079-b5ea-afa29eb58d08","Type":"ContainerStarted","Data":"3165c841189aa6e0744686ee6ee1261439cc9a472ce7ea0bf03e03473af77f46"} Oct 31 22:08:11.365049 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:11.365003 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" podStartSLOduration=1.364991017 podStartE2EDuration="1.364991017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:08:11.364015832 +0000 UTC m=+3131.489123063" watchObservedRunningTime="2026-10-31 22:08:11.364991017 +0000 UTC m=+3131.490098246" Oct 31 22:08:17.007895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.007860 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867"] Oct 31 22:08:17.013439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.013420 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.031219 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.031188 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867"] Oct 31 22:08:17.037530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.037504 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:08:17.037530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.037510 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:08:17.119323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.118441 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.119323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.118549 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dftb\" (UniqueName: \"kubernetes.io/projected/ceb59d8a-a11d-4371-a69c-de170a7dd517-kube-api-access-4dftb\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.119323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.118649 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.223339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.223238 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.223549 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.223363 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dftb\" (UniqueName: \"kubernetes.io/projected/ceb59d8a-a11d-4371-a69c-de170a7dd517-kube-api-access-4dftb\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.223549 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.223499 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.225161 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.225131 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.226978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.226959 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.247602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.247581 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dftb\" (UniqueName: \"kubernetes.io/projected/ceb59d8a-a11d-4371-a69c-de170a7dd517-kube-api-access-4dftb\") pod \"maas-api-revoke-sub-acb0e2de-nv867\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.329384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.329327 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:17.693283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.693251 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867"] Oct 31 22:08:17.694844 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:08:17.694812 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb59d8a_a11d_4371_a69c_de170a7dd517.slice/crio-ed986d6d384d6d0b5e5f896ee91d40fc6d67563b8dfa7627d8bef2f6e9aabc36 WatchSource:0}: Error finding container ed986d6d384d6d0b5e5f896ee91d40fc6d67563b8dfa7627d8bef2f6e9aabc36: Status 404 returned error can't find the container with id ed986d6d384d6d0b5e5f896ee91d40fc6d67563b8dfa7627d8bef2f6e9aabc36 Oct 31 22:08:17.948901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.948833 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs"] Oct 31 22:08:17.952603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:17.952575 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:18.373157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:18.373119 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" event={"ID":"ceb59d8a-a11d-4371-a69c-de170a7dd517","Type":"ContainerStarted","Data":"3e4aaa112f22396c10928cf47dbfd5318365074e04ca125d16f27c801ae5070c"} Oct 31 22:08:18.373604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:18.373177 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" event={"ID":"ceb59d8a-a11d-4371-a69c-de170a7dd517","Type":"ContainerStarted","Data":"ed986d6d384d6d0b5e5f896ee91d40fc6d67563b8dfa7627d8bef2f6e9aabc36"} Oct 31 22:08:18.418432 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:18.418383 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" podStartSLOduration=2.41836723 podStartE2EDuration="2.41836723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:08:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:08:18.41739218 +0000 UTC m=+3138.542499423" watchObservedRunningTime="2026-10-31 22:08:18.41836723 +0000 UTC m=+3138.543474460" Oct 31 22:08:19.381970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:19.381936 4294 generic.go:356] "Generic (PLEG): container finished" podID="ceb59d8a-a11d-4371-a69c-de170a7dd517" containerID="3e4aaa112f22396c10928cf47dbfd5318365074e04ca125d16f27c801ae5070c" exitCode=0 Oct 31 22:08:19.382449 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:19.382019 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" event={"ID":"ceb59d8a-a11d-4371-a69c-de170a7dd517","Type":"ContainerDied","Data":"3e4aaa112f22396c10928cf47dbfd5318365074e04ca125d16f27c801ae5070c"} Oct 31 22:08:20.537932 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.537907 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:20.589190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.589163 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-openshift-ca\") pod \"ceb59d8a-a11d-4371-a69c-de170a7dd517\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " Oct 31 22:08:20.589481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.589450 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb59d8a-a11d-4371-a69c-de170a7dd517-kube-api-access-4dftb\" (UniqueName: \"kubernetes.io/projected/ceb59d8a-a11d-4371-a69c-de170a7dd517-kube-api-access-4dftb\") pod \"ceb59d8a-a11d-4371-a69c-de170a7dd517\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " Oct 31 22:08:20.589646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.589500 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-openshift-ca" pod "ceb59d8a-a11d-4371-a69c-de170a7dd517" (UID: "ceb59d8a-a11d-4371-a69c-de170a7dd517"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:20.589646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.589627 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-xks-ca\") pod \"ceb59d8a-a11d-4371-a69c-de170a7dd517\" (UID: \"ceb59d8a-a11d-4371-a69c-de170a7dd517\") " Oct 31 22:08:20.590249 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.590223 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:20.592113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.592085 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb59d8a-a11d-4371-a69c-de170a7dd517-kube-api-access-4dftb" pod "ceb59d8a-a11d-4371-a69c-de170a7dd517" (UID: "ceb59d8a-a11d-4371-a69c-de170a7dd517"). InnerVolumeSpecName "kube-api-access-4dftb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:20.592508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.592487 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-xks-ca" pod "ceb59d8a-a11d-4371-a69c-de170a7dd517" (UID: "ceb59d8a-a11d-4371-a69c-de170a7dd517"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:08:20.691340 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.691255 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceb59d8a-a11d-4371-a69c-de170a7dd517-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:20.691340 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:20.691279 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dftb\" (UniqueName: \"kubernetes.io/projected/ceb59d8a-a11d-4371-a69c-de170a7dd517-kube-api-access-4dftb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:21.396279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:21.396249 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" Oct 31 22:08:21.396279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:21.396256 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867" event={"ID":"ceb59d8a-a11d-4371-a69c-de170a7dd517","Type":"ContainerDied","Data":"ed986d6d384d6d0b5e5f896ee91d40fc6d67563b8dfa7627d8bef2f6e9aabc36"} Oct 31 22:08:21.396569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:21.396327 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed986d6d384d6d0b5e5f896ee91d40fc6d67563b8dfa7627d8bef2f6e9aabc36" Oct 31 22:08:22.949539 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:22.949498 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" podUID="a3510a35-28bc-4079-b5ea-afa29eb58d08" containerName="payload-processing" containerID="cri-o://a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444" gracePeriod=25 Oct 31 22:08:23.246618 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.246595 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:23.324059 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324029 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-kube-api-access-cm4m9\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-kube-api-access-cm4m9\") pod \"a3510a35-28bc-4079-b5ea-afa29eb58d08\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " Oct 31 22:08:23.324259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324070 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-plugins-config-volume\") pod \"a3510a35-28bc-4079-b5ea-afa29eb58d08\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " Oct 31 22:08:23.324259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324103 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-routing-overlay-volume\") pod \"a3510a35-28bc-4079-b5ea-afa29eb58d08\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " Oct 31 22:08:23.324259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324198 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-provider-credentials\") pod \"a3510a35-28bc-4079-b5ea-afa29eb58d08\" (UID: \"a3510a35-28bc-4079-b5ea-afa29eb58d08\") " Oct 31 22:08:23.324970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324551 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-routing-overlay-volume" pod "a3510a35-28bc-4079-b5ea-afa29eb58d08" (UID: "a3510a35-28bc-4079-b5ea-afa29eb58d08"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:23.324970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324571 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-plugins-config-volume" pod "a3510a35-28bc-4079-b5ea-afa29eb58d08" (UID: "a3510a35-28bc-4079-b5ea-afa29eb58d08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:23.324970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324920 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:23.324970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.324938 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a3510a35-28bc-4079-b5ea-afa29eb58d08-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:23.326472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.326439 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-kube-api-access-cm4m9" pod "a3510a35-28bc-4079-b5ea-afa29eb58d08" (UID: "a3510a35-28bc-4079-b5ea-afa29eb58d08"). InnerVolumeSpecName "kube-api-access-cm4m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:23.326595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.326516 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-provider-credentials" pod "a3510a35-28bc-4079-b5ea-afa29eb58d08" (UID: "a3510a35-28bc-4079-b5ea-afa29eb58d08"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:23.414275 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.414237 4294 generic.go:356] "Generic (PLEG): container finished" podID="a3510a35-28bc-4079-b5ea-afa29eb58d08" containerID="a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444" exitCode=0 Oct 31 22:08:23.414458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.414288 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" Oct 31 22:08:23.414458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.414357 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" event={"ID":"a3510a35-28bc-4079-b5ea-afa29eb58d08","Type":"ContainerDied","Data":"a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444"} Oct 31 22:08:23.414458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.414428 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs" event={"ID":"a3510a35-28bc-4079-b5ea-afa29eb58d08","Type":"ContainerDied","Data":"3165c841189aa6e0744686ee6ee1261439cc9a472ce7ea0bf03e03473af77f46"} Oct 31 22:08:23.414622 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.414465 4294 scope.go:122] "RemoveContainer" containerID="a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444" Oct 31 22:08:23.424944 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.424930 4294 scope.go:122] "RemoveContainer" containerID="a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444" Oct 31 22:08:23.425176 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:08:23.425156 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444\": container with ID starting with a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444 not found: ID does not exist" containerID="a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444" Oct 31 22:08:23.425256 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.425182 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444"} err="failed to get container status \"a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444\": rpc error: code = NotFound desc = could not find container \"a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444\": container with ID starting with a7daa45ce7ddaa8f78054f7dcbeb4a137395708e1db4c476da6d04c4a560d444 not found: ID does not exist" Oct 31 22:08:23.425780 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.425759 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm4m9\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-kube-api-access-cm4m9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:23.425824 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.425784 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a3510a35-28bc-4079-b5ea-afa29eb58d08-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:23.469862 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.469804 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs"] Oct 31 22:08:23.473281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.473258 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-696f9a/payload-processing-external-model-e2e-extmod-a-696f9a-9c564c4fs"] Oct 31 22:08:23.986250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:23.986218 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3510a35-28bc-4079-b5ea-afa29eb58d08" path="/var/lib/kubelet/pods/a3510a35-28bc-4079-b5ea-afa29eb58d08/volumes" Oct 31 22:08:27.674128 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.674092 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm"] Oct 31 22:08:27.674946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.674923 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a3510a35-28bc-4079-b5ea-afa29eb58d08" containerName="payload-processing" Oct 31 22:08:27.674946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.674947 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3510a35-28bc-4079-b5ea-afa29eb58d08" containerName="payload-processing" Oct 31 22:08:27.675117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.674997 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ceb59d8a-a11d-4371-a69c-de170a7dd517" containerName="revoke-keys" Oct 31 22:08:27.675117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.675005 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb59d8a-a11d-4371-a69c-de170a7dd517" containerName="revoke-keys" Oct 31 22:08:27.675231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.675156 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3510a35-28bc-4079-b5ea-afa29eb58d08" containerName="payload-processing" Oct 31 22:08:27.675231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.675180 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb59d8a-a11d-4371-a69c-de170a7dd517" containerName="revoke-keys" Oct 31 22:08:27.680142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.680122 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.695134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.695110 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm"] Oct 31 22:08:27.703379 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.703355 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:08:27.703575 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.703555 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:08:27.775782 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.775748 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.775952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.775792 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.775952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.775872 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbdqd\" (UniqueName: \"kubernetes.io/projected/bbf967bf-61f1-4176-8afb-ec496d753060-kube-api-access-gbdqd\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.877118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.877076 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.877279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.877130 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.877279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.877173 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbdqd\" (UniqueName: \"kubernetes.io/projected/bbf967bf-61f1-4176-8afb-ec496d753060-kube-api-access-gbdqd\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.877869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.877843 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.879351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.879332 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.900566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.900540 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbdqd\" (UniqueName: \"kubernetes.io/projected/bbf967bf-61f1-4176-8afb-ec496d753060-kube-api-access-gbdqd\") pod \"maas-api-revoke-sub-3e5b549e-4wjsm\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:27.993649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:27.993624 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:28.312367 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:28.311632 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm"] Oct 31 22:08:28.321644 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:08:28.321616 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf967bf_61f1_4176_8afb_ec496d753060.slice/crio-806ec28a2a520fa204f22085fdcbd4e460c2e09072049158298de3e86ed552ee WatchSource:0}: Error finding container 806ec28a2a520fa204f22085fdcbd4e460c2e09072049158298de3e86ed552ee: Status 404 returned error can't find the container with id 806ec28a2a520fa204f22085fdcbd4e460c2e09072049158298de3e86ed552ee Oct 31 22:08:28.450656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:28.450621 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" event={"ID":"bbf967bf-61f1-4176-8afb-ec496d753060","Type":"ContainerStarted","Data":"806ec28a2a520fa204f22085fdcbd4e460c2e09072049158298de3e86ed552ee"} Oct 31 22:08:29.458787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:29.458710 4294 generic.go:356] "Generic (PLEG): container finished" podID="bbf967bf-61f1-4176-8afb-ec496d753060" containerID="f3badaf30d2a01749150ae2ac0da30f59b0a9e8153bbeace492dcca59d247e36" exitCode=0 Oct 31 22:08:29.458787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:29.458757 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" event={"ID":"bbf967bf-61f1-4176-8afb-ec496d753060","Type":"ContainerDied","Data":"f3badaf30d2a01749150ae2ac0da30f59b0a9e8153bbeace492dcca59d247e36"} Oct 31 22:08:30.621175 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.621148 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:30.714365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.714337 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-openshift-ca\") pod \"bbf967bf-61f1-4176-8afb-ec496d753060\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " Oct 31 22:08:30.714517 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.714484 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-xks-ca\") pod \"bbf967bf-61f1-4176-8afb-ec496d753060\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " Oct 31 22:08:30.714656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.714644 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbf967bf-61f1-4176-8afb-ec496d753060-kube-api-access-gbdqd\" (UniqueName: \"kubernetes.io/projected/bbf967bf-61f1-4176-8afb-ec496d753060-kube-api-access-gbdqd\") pod \"bbf967bf-61f1-4176-8afb-ec496d753060\" (UID: \"bbf967bf-61f1-4176-8afb-ec496d753060\") " Oct 31 22:08:30.714724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.714669 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-openshift-ca" pod "bbf967bf-61f1-4176-8afb-ec496d753060" (UID: "bbf967bf-61f1-4176-8afb-ec496d753060"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:30.715109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.715091 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:30.716563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.716511 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-xks-ca" pod "bbf967bf-61f1-4176-8afb-ec496d753060" (UID: "bbf967bf-61f1-4176-8afb-ec496d753060"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:08:30.716670 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.716650 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf967bf-61f1-4176-8afb-ec496d753060-kube-api-access-gbdqd" pod "bbf967bf-61f1-4176-8afb-ec496d753060" (UID: "bbf967bf-61f1-4176-8afb-ec496d753060"). InnerVolumeSpecName "kube-api-access-gbdqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:30.815917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.815878 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbf967bf-61f1-4176-8afb-ec496d753060-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:30.815917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:30.815914 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbdqd\" (UniqueName: \"kubernetes.io/projected/bbf967bf-61f1-4176-8afb-ec496d753060-kube-api-access-gbdqd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:31.033163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:31.033134 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf"] Oct 31 22:08:31.036512 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:31.036490 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8964a30f-w7jcf"] Oct 31 22:08:31.310708 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:08:31.310624 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:08:31.476170 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:31.476128 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" event={"ID":"bbf967bf-61f1-4176-8afb-ec496d753060","Type":"ContainerDied","Data":"806ec28a2a520fa204f22085fdcbd4e460c2e09072049158298de3e86ed552ee"} Oct 31 22:08:31.476398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:31.476190 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="806ec28a2a520fa204f22085fdcbd4e460c2e09072049158298de3e86ed552ee" Oct 31 22:08:31.476398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:31.476259 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm" Oct 31 22:08:31.987601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:31.987567 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7798a306-9b5b-450c-a3cc-9f9ec3b588ef" path="/var/lib/kubelet/pods/7798a306-9b5b-450c-a3cc-9f9ec3b588ef/volumes" Oct 31 22:08:39.890802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.890770 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt"] Oct 31 22:08:39.891487 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.891470 4294 cpu_manager.go:414] "containerMap: removing container" podUID="bbf967bf-61f1-4176-8afb-ec496d753060" containerName="revoke-keys" Oct 31 22:08:39.891551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.891488 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbf967bf-61f1-4176-8afb-ec496d753060" containerName="revoke-keys" Oct 31 22:08:39.891596 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.891587 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbf967bf-61f1-4176-8afb-ec496d753060" containerName="revoke-keys" Oct 31 22:08:39.895248 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.895232 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:39.912593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.912569 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt"] Oct 31 22:08:39.920230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.920208 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:08:39.920362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.920353 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:08:40.000328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:39.967681 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.046491 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.046454 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mrdf\" (UniqueName: \"kubernetes.io/projected/aacb9938-e4e1-4e28-918e-0a6b26cdf014-kube-api-access-5mrdf\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.047007 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.046986 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.163388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.157503 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.163388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.157701 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.163388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.158177 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mrdf\" (UniqueName: \"kubernetes.io/projected/aacb9938-e4e1-4e28-918e-0a6b26cdf014-kube-api-access-5mrdf\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.163388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.158432 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.163388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.160428 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.181408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.181381 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mrdf\" (UniqueName: \"kubernetes.io/projected/aacb9938-e4e1-4e28-918e-0a6b26cdf014-kube-api-access-5mrdf\") pod \"maas-api-revoke-sub-f3735e7a-wkgtt\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.209264 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.209237 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:40.557346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.552388 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt"] Oct 31 22:08:40.681522 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.681490 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7"] Oct 31 22:08:40.686841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.686823 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.703866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.703835 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7"] Oct 31 22:08:40.780255 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.780227 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.780407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.780297 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.780470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.780453 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhht\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-kube-api-access-9qhht\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.780518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.780506 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.881714 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.881677 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.881825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.881789 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.881909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.881890 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhht\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-kube-api-access-9qhht\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.881967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.881956 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.882500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.882474 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.882690 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.882665 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.885061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.885019 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:40.905918 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:40.905899 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhht\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-kube-api-access-9qhht\") pod \"payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:41.004913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.004892 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:41.034147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.034125 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j"] Oct 31 22:08:41.037697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.037673 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77fecb57-r5x4j"] Oct 31 22:08:41.307089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.307054 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7"] Oct 31 22:08:41.307741 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:08:41.307714 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43a7059_9745_494a_920c_4147e4d9625d.slice/crio-d6c5bfc0b7f0c95c639813efb366081d9029f5d1a2904367a5bd2a7daf27282a WatchSource:0}: Error finding container d6c5bfc0b7f0c95c639813efb366081d9029f5d1a2904367a5bd2a7daf27282a: Status 404 returned error can't find the container with id d6c5bfc0b7f0c95c639813efb366081d9029f5d1a2904367a5bd2a7daf27282a Oct 31 22:08:41.560552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.560481 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" event={"ID":"d43a7059-9745-494a-920c-4147e4d9625d","Type":"ContainerStarted","Data":"ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675"} Oct 31 22:08:41.560552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.560539 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" event={"ID":"d43a7059-9745-494a-920c-4147e4d9625d","Type":"ContainerStarted","Data":"d6c5bfc0b7f0c95c639813efb366081d9029f5d1a2904367a5bd2a7daf27282a"} Oct 31 22:08:41.560739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.560569 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:41.562176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.562152 4294 generic.go:356] "Generic (PLEG): container finished" podID="aacb9938-e4e1-4e28-918e-0a6b26cdf014" containerID="1f3753a46fabe8545a30a8a90f319a598de03f8ce63d281c07f8ca082de7ba29" exitCode=0 Oct 31 22:08:41.562261 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.562211 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" event={"ID":"aacb9938-e4e1-4e28-918e-0a6b26cdf014","Type":"ContainerDied","Data":"1f3753a46fabe8545a30a8a90f319a598de03f8ce63d281c07f8ca082de7ba29"} Oct 31 22:08:41.562261 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.562252 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" event={"ID":"aacb9938-e4e1-4e28-918e-0a6b26cdf014","Type":"ContainerStarted","Data":"336a8710f852e7b26f91a26d444976e3adce9ea811c4d7f665383d09780f8a8b"} Oct 31 22:08:41.602766 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.602722 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" podStartSLOduration=1.6027066429999999 podStartE2EDuration="1.602706643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:08:41.601789844 +0000 UTC m=+3161.726897075" watchObservedRunningTime="2026-10-31 22:08:41.602706643 +0000 UTC m=+3161.727813872" Oct 31 22:08:41.988003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:41.987968 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe" path="/var/lib/kubelet/pods/e5f0e45b-f1ca-407d-a3fa-d6e3af734bfe/volumes" Oct 31 22:08:42.730018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.729992 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:42.810415 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.810386 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-xks-ca\") pod \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " Oct 31 22:08:42.810607 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.810477 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-openshift-ca\") pod \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " Oct 31 22:08:42.810607 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.810537 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacb9938-e4e1-4e28-918e-0a6b26cdf014-kube-api-access-5mrdf\" (UniqueName: \"kubernetes.io/projected/aacb9938-e4e1-4e28-918e-0a6b26cdf014-kube-api-access-5mrdf\") pod \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\" (UID: \"aacb9938-e4e1-4e28-918e-0a6b26cdf014\") " Oct 31 22:08:42.810851 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.810823 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-openshift-ca" pod "aacb9938-e4e1-4e28-918e-0a6b26cdf014" (UID: "aacb9938-e4e1-4e28-918e-0a6b26cdf014"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:42.811254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.811234 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:42.812658 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.812634 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacb9938-e4e1-4e28-918e-0a6b26cdf014-kube-api-access-5mrdf" pod "aacb9938-e4e1-4e28-918e-0a6b26cdf014" (UID: "aacb9938-e4e1-4e28-918e-0a6b26cdf014"). InnerVolumeSpecName "kube-api-access-5mrdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:42.813003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.812988 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-xks-ca" pod "aacb9938-e4e1-4e28-918e-0a6b26cdf014" (UID: "aacb9938-e4e1-4e28-918e-0a6b26cdf014"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:08:42.912227 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.912200 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacb9938-e4e1-4e28-918e-0a6b26cdf014-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:42.912227 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:42.912227 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mrdf\" (UniqueName: \"kubernetes.io/projected/aacb9938-e4e1-4e28-918e-0a6b26cdf014-kube-api-access-5mrdf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:43.581474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:43.581442 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" Oct 31 22:08:43.581889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:43.581522 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt" event={"ID":"aacb9938-e4e1-4e28-918e-0a6b26cdf014","Type":"ContainerDied","Data":"336a8710f852e7b26f91a26d444976e3adce9ea811c4d7f665383d09780f8a8b"} Oct 31 22:08:43.581889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:43.581574 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336a8710f852e7b26f91a26d444976e3adce9ea811c4d7f665383d09780f8a8b" Oct 31 22:08:50.667311 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.667268 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg"] Oct 31 22:08:50.667950 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.667936 4294 cpu_manager.go:414] "containerMap: removing container" podUID="aacb9938-e4e1-4e28-918e-0a6b26cdf014" containerName="revoke-keys" Oct 31 22:08:50.668402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.667951 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacb9938-e4e1-4e28-918e-0a6b26cdf014" containerName="revoke-keys" Oct 31 22:08:50.668402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.668057 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacb9938-e4e1-4e28-918e-0a6b26cdf014" containerName="revoke-keys" Oct 31 22:08:50.671818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.671801 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.689693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.689665 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg"] Oct 31 22:08:50.696719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.696695 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:08:50.699525 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.699500 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:08:50.814249 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.814216 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.814474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.814288 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.814474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.814415 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-966nt\" (UniqueName: \"kubernetes.io/projected/b220b73a-fc08-4891-b89e-4e78904b2510-kube-api-access-966nt\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.916423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.916391 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-966nt\" (UniqueName: \"kubernetes.io/projected/b220b73a-fc08-4891-b89e-4e78904b2510-kube-api-access-966nt\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.916678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.916661 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.916766 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.916752 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.917284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.917263 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.919070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.919022 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.939733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.939706 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-966nt\" (UniqueName: \"kubernetes.io/projected/b220b73a-fc08-4891-b89e-4e78904b2510-kube-api-access-966nt\") pod \"maas-api-revoke-sub-cc5756d5-zlggg\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:50.985060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:50.985026 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:51.287940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:51.287906 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg"] Oct 31 22:08:51.288783 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:08:51.288753 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb220b73a_fc08_4891_b89e_4e78904b2510.slice/crio-4bab8a35e9820d02fef83880cee4dd0710a913fe54e0174a5349592eacdc67ef WatchSource:0}: Error finding container 4bab8a35e9820d02fef83880cee4dd0710a913fe54e0174a5349592eacdc67ef: Status 404 returned error can't find the container with id 4bab8a35e9820d02fef83880cee4dd0710a913fe54e0174a5349592eacdc67ef Oct 31 22:08:51.642604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:51.642522 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" event={"ID":"b220b73a-fc08-4891-b89e-4e78904b2510","Type":"ContainerStarted","Data":"4bab8a35e9820d02fef83880cee4dd0710a913fe54e0174a5349592eacdc67ef"} Oct 31 22:08:52.572125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:52.572092 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:08:52.649367 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:52.649326 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx"] Oct 31 22:08:52.657173 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:52.657137 4294 generic.go:356] "Generic (PLEG): container finished" podID="b220b73a-fc08-4891-b89e-4e78904b2510" containerID="5f7d1db76ebb2c997ecd5aedb8f7e4c8d70a9edea17115d371e034c846dabe3a" exitCode=0 Oct 31 22:08:52.657360 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:52.657211 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" event={"ID":"b220b73a-fc08-4891-b89e-4e78904b2510","Type":"ContainerDied","Data":"5f7d1db76ebb2c997ecd5aedb8f7e4c8d70a9edea17115d371e034c846dabe3a"} Oct 31 22:08:53.842633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:53.842604 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:53.961884 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:53.961786 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-xks-ca\") pod \"b220b73a-fc08-4891-b89e-4e78904b2510\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " Oct 31 22:08:53.962068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:53.961935 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b220b73a-fc08-4891-b89e-4e78904b2510-kube-api-access-966nt\" (UniqueName: \"kubernetes.io/projected/b220b73a-fc08-4891-b89e-4e78904b2510-kube-api-access-966nt\") pod \"b220b73a-fc08-4891-b89e-4e78904b2510\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " Oct 31 22:08:53.962068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:53.961978 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-openshift-ca\") pod \"b220b73a-fc08-4891-b89e-4e78904b2510\" (UID: \"b220b73a-fc08-4891-b89e-4e78904b2510\") " Oct 31 22:08:53.962765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:53.962733 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-openshift-ca" pod "b220b73a-fc08-4891-b89e-4e78904b2510" (UID: "b220b73a-fc08-4891-b89e-4e78904b2510"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:53.965331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:53.965288 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b220b73a-fc08-4891-b89e-4e78904b2510-kube-api-access-966nt" pod "b220b73a-fc08-4891-b89e-4e78904b2510" (UID: "b220b73a-fc08-4891-b89e-4e78904b2510"). InnerVolumeSpecName "kube-api-access-966nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:53.966658 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:53.966632 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-xks-ca" pod "b220b73a-fc08-4891-b89e-4e78904b2510" (UID: "b220b73a-fc08-4891-b89e-4e78904b2510"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:08:54.063850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:54.063817 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:54.063850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:54.063844 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-966nt\" (UniqueName: \"kubernetes.io/projected/b220b73a-fc08-4891-b89e-4e78904b2510-kube-api-access-966nt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:54.063850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:54.063855 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b220b73a-fc08-4891-b89e-4e78904b2510-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:54.672448 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:54.672420 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" Oct 31 22:08:54.672644 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:54.672452 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg" event={"ID":"b220b73a-fc08-4891-b89e-4e78904b2510","Type":"ContainerDied","Data":"4bab8a35e9820d02fef83880cee4dd0710a913fe54e0174a5349592eacdc67ef"} Oct 31 22:08:54.672644 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:54.672508 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bab8a35e9820d02fef83880cee4dd0710a913fe54e0174a5349592eacdc67ef" Oct 31 22:08:57.650109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:57.650053 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" podUID="ff335783-0ce6-410b-b80e-209988d7ebea" containerName="payload-processing" containerID="cri-o://a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04" gracePeriod=25 Oct 31 22:08:58.040044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.040018 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:08:58.134882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.134840 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-kube-api-access-7r4xf\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-kube-api-access-7r4xf\") pod \"ff335783-0ce6-410b-b80e-209988d7ebea\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " Oct 31 22:08:58.135095 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.134965 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-provider-credentials\") pod \"ff335783-0ce6-410b-b80e-209988d7ebea\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " Oct 31 22:08:58.135174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.135142 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-plugins-config-volume\") pod \"ff335783-0ce6-410b-b80e-209988d7ebea\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " Oct 31 22:08:58.135262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.135240 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-routing-overlay-volume\") pod \"ff335783-0ce6-410b-b80e-209988d7ebea\" (UID: \"ff335783-0ce6-410b-b80e-209988d7ebea\") " Oct 31 22:08:58.135753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.135721 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-plugins-config-volume" pod "ff335783-0ce6-410b-b80e-209988d7ebea" (UID: "ff335783-0ce6-410b-b80e-209988d7ebea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:58.135856 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.135814 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-routing-overlay-volume" pod "ff335783-0ce6-410b-b80e-209988d7ebea" (UID: "ff335783-0ce6-410b-b80e-209988d7ebea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:08:58.136398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.136358 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:58.136398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.136389 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff335783-0ce6-410b-b80e-209988d7ebea-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:58.137821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.137794 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-kube-api-access-7r4xf" pod "ff335783-0ce6-410b-b80e-209988d7ebea" (UID: "ff335783-0ce6-410b-b80e-209988d7ebea"). InnerVolumeSpecName "kube-api-access-7r4xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:58.140811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.140764 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-provider-credentials" pod "ff335783-0ce6-410b-b80e-209988d7ebea" (UID: "ff335783-0ce6-410b-b80e-209988d7ebea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:08:58.238028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.237996 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r4xf\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-kube-api-access-7r4xf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:58.238028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.238026 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff335783-0ce6-410b-b80e-209988d7ebea-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:08:58.768381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.766951 4294 generic.go:356] "Generic (PLEG): container finished" podID="ff335783-0ce6-410b-b80e-209988d7ebea" containerID="a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04" exitCode=0 Oct 31 22:08:58.768381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.767021 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" event={"ID":"ff335783-0ce6-410b-b80e-209988d7ebea","Type":"ContainerDied","Data":"a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04"} Oct 31 22:08:58.768381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.767071 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" event={"ID":"ff335783-0ce6-410b-b80e-209988d7ebea","Type":"ContainerDied","Data":"a0617c001145ec63c337cc591655ee4758eb62274e0d7829d933ed14f7ca8f7f"} Oct 31 22:08:58.768381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.767108 4294 scope.go:122] "RemoveContainer" containerID="a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04" Oct 31 22:08:58.768381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.767273 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx" Oct 31 22:08:58.861492 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.861027 4294 scope.go:122] "RemoveContainer" containerID="a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04" Oct 31 22:08:58.861895 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:08:58.861792 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04\": container with ID starting with a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04 not found: ID does not exist" containerID="a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04" Oct 31 22:08:58.861895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.861826 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04"} err="failed to get container status \"a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04\": rpc error: code = NotFound desc = could not find container \"a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04\": container with ID starting with a3502fd7e72474586e5ebbe1cac9834daeea12b752fbb2dc7f6a6c9d05175c04 not found: ID does not exist" Oct 31 22:08:58.880445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.880398 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx"] Oct 31 22:08:58.915810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:58.915779 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-5c478kvvx"] Oct 31 22:08:59.985775 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:08:59.985733 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff335783-0ce6-410b-b80e-209988d7ebea" path="/var/lib/kubelet/pods/ff335783-0ce6-410b-b80e-209988d7ebea/volumes" Oct 31 22:09:01.311334 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:09:01.311271 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:09:02.321356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.321283 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl"] Oct 31 22:09:02.321966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.321949 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ff335783-0ce6-410b-b80e-209988d7ebea" containerName="payload-processing" Oct 31 22:09:02.322061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.321975 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff335783-0ce6-410b-b80e-209988d7ebea" containerName="payload-processing" Oct 31 22:09:02.322061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.322002 4294 cpu_manager.go:414] "containerMap: removing container" podUID="b220b73a-fc08-4891-b89e-4e78904b2510" containerName="revoke-keys" Oct 31 22:09:02.322061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.322007 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b220b73a-fc08-4891-b89e-4e78904b2510" containerName="revoke-keys" Oct 31 22:09:02.322230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.322131 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff335783-0ce6-410b-b80e-209988d7ebea" containerName="payload-processing" Oct 31 22:09:02.322230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.322142 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b220b73a-fc08-4891-b89e-4e78904b2510" containerName="revoke-keys" Oct 31 22:09:02.327222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.327197 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.346407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.346365 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl"] Oct 31 22:09:02.351529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.351504 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:09:02.351885 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.351846 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:09:02.434077 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.434035 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.434359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.434207 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.434359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.434264 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p8xn\" (UniqueName: \"kubernetes.io/projected/71e858c6-9c71-46c1-9fa4-e59a993604a8-kube-api-access-9p8xn\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.535738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.535699 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.535922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.535794 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.535922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.535826 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p8xn\" (UniqueName: \"kubernetes.io/projected/71e858c6-9c71-46c1-9fa4-e59a993604a8-kube-api-access-9p8xn\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.536524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.536501 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.538189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.538168 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.580882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.580802 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p8xn\" (UniqueName: \"kubernetes.io/projected/71e858c6-9c71-46c1-9fa4-e59a993604a8-kube-api-access-9p8xn\") pod \"maas-api-revoke-sub-04d6e260-6btkl\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:02.645589 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:02.645550 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:03.021388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:03.021348 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl"] Oct 31 22:09:03.022352 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:03.022296 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e858c6_9c71_46c1_9fa4_e59a993604a8.slice/crio-581f13023fc23aca778f31d4af9a19820facc8b14196347c70f7692830479330 WatchSource:0}: Error finding container 581f13023fc23aca778f31d4af9a19820facc8b14196347c70f7692830479330: Status 404 returned error can't find the container with id 581f13023fc23aca778f31d4af9a19820facc8b14196347c70f7692830479330 Oct 31 22:09:03.031765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:03.031740 4294 scope.go:122] "RemoveContainer" containerID="7857c5f462c369caf8183119bc01ac11622a6a7684ca4043071be553dd27b49c" Oct 31 22:09:03.041651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:03.041627 4294 scope.go:122] "RemoveContainer" containerID="0c81e5cc345da57662f8f2c685489a662e2ee93f232ced9479409aaf3f338b55" Oct 31 22:09:03.180727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:03.180699 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7"] Oct 31 22:09:03.804367 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:03.804327 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" event={"ID":"71e858c6-9c71-46c1-9fa4-e59a993604a8","Type":"ContainerStarted","Data":"07981f1b75b8c9bb4e838804384ac1a21ef3d6292f3cc24538d377a22c7c6d46"} Oct 31 22:09:03.804764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:03.804388 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" event={"ID":"71e858c6-9c71-46c1-9fa4-e59a993604a8","Type":"ContainerStarted","Data":"581f13023fc23aca778f31d4af9a19820facc8b14196347c70f7692830479330"} Oct 31 22:09:03.846857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:03.846802 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" podStartSLOduration=1.846788131 podStartE2EDuration="1.846788131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:03.844694473 +0000 UTC m=+3183.969801704" watchObservedRunningTime="2026-10-31 22:09:03.846788131 +0000 UTC m=+3183.971895387" Oct 31 22:09:04.812880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:04.812848 4294 generic.go:356] "Generic (PLEG): container finished" podID="71e858c6-9c71-46c1-9fa4-e59a993604a8" containerID="07981f1b75b8c9bb4e838804384ac1a21ef3d6292f3cc24538d377a22c7c6d46" exitCode=0 Oct 31 22:09:04.813352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:04.812932 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" event={"ID":"71e858c6-9c71-46c1-9fa4-e59a993604a8","Type":"ContainerDied","Data":"07981f1b75b8c9bb4e838804384ac1a21ef3d6292f3cc24538d377a22c7c6d46"} Oct 31 22:09:05.965723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:05.965696 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:06.012406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.012380 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-openshift-ca\") pod \"71e858c6-9c71-46c1-9fa4-e59a993604a8\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " Oct 31 22:09:06.012703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.012671 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e858c6-9c71-46c1-9fa4-e59a993604a8-kube-api-access-9p8xn\" (UniqueName: \"kubernetes.io/projected/71e858c6-9c71-46c1-9fa4-e59a993604a8-kube-api-access-9p8xn\") pod \"71e858c6-9c71-46c1-9fa4-e59a993604a8\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " Oct 31 22:09:06.012799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.012709 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-xks-ca\") pod \"71e858c6-9c71-46c1-9fa4-e59a993604a8\" (UID: \"71e858c6-9c71-46c1-9fa4-e59a993604a8\") " Oct 31 22:09:06.013595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.013568 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-openshift-ca" pod "71e858c6-9c71-46c1-9fa4-e59a993604a8" (UID: "71e858c6-9c71-46c1-9fa4-e59a993604a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:06.016470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.016449 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-xks-ca" pod "71e858c6-9c71-46c1-9fa4-e59a993604a8" (UID: "71e858c6-9c71-46c1-9fa4-e59a993604a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:09:06.016614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.016484 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e858c6-9c71-46c1-9fa4-e59a993604a8-kube-api-access-9p8xn" pod "71e858c6-9c71-46c1-9fa4-e59a993604a8" (UID: "71e858c6-9c71-46c1-9fa4-e59a993604a8"). InnerVolumeSpecName "kube-api-access-9p8xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:06.114815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.114743 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p8xn\" (UniqueName: \"kubernetes.io/projected/71e858c6-9c71-46c1-9fa4-e59a993604a8-kube-api-access-9p8xn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:06.114815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.114773 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:06.114815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.114781 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e858c6-9c71-46c1-9fa4-e59a993604a8-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:06.828445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.828416 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" Oct 31 22:09:06.828609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.828455 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl" event={"ID":"71e858c6-9c71-46c1-9fa4-e59a993604a8","Type":"ContainerDied","Data":"581f13023fc23aca778f31d4af9a19820facc8b14196347c70f7692830479330"} Oct 31 22:09:06.828609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:06.828508 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581f13023fc23aca778f31d4af9a19820facc8b14196347c70f7692830479330" Oct 31 22:09:08.181608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.181565 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" podUID="d43a7059-9745-494a-920c-4147e4d9625d" containerName="payload-processing" containerID="cri-o://ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675" gracePeriod=25 Oct 31 22:09:08.469630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.469605 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:09:08.545081 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545041 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-routing-overlay-volume\") pod \"d43a7059-9745-494a-920c-4147e4d9625d\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " Oct 31 22:09:08.545284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545260 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-plugins-config-volume\") pod \"d43a7059-9745-494a-920c-4147e4d9625d\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " Oct 31 22:09:08.545418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545400 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-kube-api-access-9qhht\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-kube-api-access-9qhht\") pod \"d43a7059-9745-494a-920c-4147e4d9625d\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " Oct 31 22:09:08.545493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545442 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-provider-credentials\") pod \"d43a7059-9745-494a-920c-4147e4d9625d\" (UID: \"d43a7059-9745-494a-920c-4147e4d9625d\") " Oct 31 22:09:08.545589 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545552 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-routing-overlay-volume" pod "d43a7059-9745-494a-920c-4147e4d9625d" (UID: "d43a7059-9745-494a-920c-4147e4d9625d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:08.545743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545718 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-plugins-config-volume" pod "d43a7059-9745-494a-920c-4147e4d9625d" (UID: "d43a7059-9745-494a-920c-4147e4d9625d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:08.545990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545972 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:08.545990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.545990 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d43a7059-9745-494a-920c-4147e4d9625d-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:08.547617 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.547592 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-provider-credentials" pod "d43a7059-9745-494a-920c-4147e4d9625d" (UID: "d43a7059-9745-494a-920c-4147e4d9625d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:08.547978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.547960 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-kube-api-access-9qhht" pod "d43a7059-9745-494a-920c-4147e4d9625d" (UID: "d43a7059-9745-494a-920c-4147e4d9625d"). InnerVolumeSpecName "kube-api-access-9qhht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:08.647926 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.647892 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhht\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-kube-api-access-9qhht\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:08.647926 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.647922 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d43a7059-9745-494a-920c-4147e4d9625d-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:08.843533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.843499 4294 generic.go:356] "Generic (PLEG): container finished" podID="d43a7059-9745-494a-920c-4147e4d9625d" containerID="ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675" exitCode=0 Oct 31 22:09:08.843740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.843571 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" Oct 31 22:09:08.843740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.843589 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" event={"ID":"d43a7059-9745-494a-920c-4147e4d9625d","Type":"ContainerDied","Data":"ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675"} Oct 31 22:09:08.843740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.843667 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7" event={"ID":"d43a7059-9745-494a-920c-4147e4d9625d","Type":"ContainerDied","Data":"d6c5bfc0b7f0c95c639813efb366081d9029f5d1a2904367a5bd2a7daf27282a"} Oct 31 22:09:08.843740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.843692 4294 scope.go:122] "RemoveContainer" containerID="ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675" Oct 31 22:09:08.854445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.854426 4294 scope.go:122] "RemoveContainer" containerID="ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675" Oct 31 22:09:08.854737 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:09:08.854708 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675\": container with ID starting with ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675 not found: ID does not exist" containerID="ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675" Oct 31 22:09:08.854810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.854745 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675"} err="failed to get container status \"ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675\": rpc error: code = NotFound desc = could not find container \"ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675\": container with ID starting with ce99442b4f0de0179ff390efb9620349f22b79af2dc6b31269cb0cf49480a675 not found: ID does not exist" Oct 31 22:09:08.896080 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.896055 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7"] Oct 31 22:09:08.902234 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:08.902212 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4a73b1/payload-processing-external-model-e2e-extmod-b-4a73b1-7f6667cv7"] Oct 31 22:09:09.987206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:09.987174 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43a7059-9745-494a-920c-4147e4d9625d" path="/var/lib/kubelet/pods/d43a7059-9745-494a-920c-4147e4d9625d/volumes" Oct 31 22:09:13.932524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:13.927353 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft"] Oct 31 22:09:13.932524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:13.930116 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d43a7059-9745-494a-920c-4147e4d9625d" containerName="payload-processing" Oct 31 22:09:13.932524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:13.930140 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43a7059-9745-494a-920c-4147e4d9625d" containerName="payload-processing" Oct 31 22:09:13.932524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:13.930171 4294 cpu_manager.go:414] "containerMap: removing container" podUID="71e858c6-9c71-46c1-9fa4-e59a993604a8" containerName="revoke-keys" Oct 31 22:09:13.932524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:13.930180 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e858c6-9c71-46c1-9fa4-e59a993604a8" containerName="revoke-keys" Oct 31 22:09:13.956650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:13.956520 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43a7059-9745-494a-920c-4147e4d9625d" containerName="payload-processing" Oct 31 22:09:13.956650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:13.956554 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e858c6-9c71-46c1-9fa4-e59a993604a8" containerName="revoke-keys" Oct 31 22:09:14.019600 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.019570 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.028960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.028927 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft"] Oct 31 22:09:14.137322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.137091 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjsdk\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-kube-api-access-mjsdk\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.137322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.137142 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-provider-credentials\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.137322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.137237 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-plugins-config-volume\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.137641 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.137447 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-routing-overlay-volume\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.238853 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.238820 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-routing-overlay-volume\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.239063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.238938 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjsdk\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-kube-api-access-mjsdk\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.239063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.238964 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-provider-credentials\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.239063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.239037 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-plugins-config-volume\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.239674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.239646 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-routing-overlay-volume\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.239931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.239910 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-plugins-config-volume\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.241506 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.241483 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-provider-credentials\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.262449 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.262421 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjsdk\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-kube-api-access-mjsdk\") pod \"payload-processing-external-model-7f548df5c7-g5nft\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.334527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.334500 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:14.639125 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:14.639094 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc139d51d_9d69_4b93_a195_4a8d74dbcabb.slice/crio-5e20318e8233a54ace88115f370f4bb591b5990bc88a6791e3406b0f15dfb070 WatchSource:0}: Error finding container 5e20318e8233a54ace88115f370f4bb591b5990bc88a6791e3406b0f15dfb070: Status 404 returned error can't find the container with id 5e20318e8233a54ace88115f370f4bb591b5990bc88a6791e3406b0f15dfb070 Oct 31 22:09:14.639500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:14.639473 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft"] Oct 31 22:09:15.017970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.017931 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" event={"ID":"c139d51d-9d69-4b93-a195-4a8d74dbcabb","Type":"ContainerStarted","Data":"f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b"} Oct 31 22:09:15.018386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.017994 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" event={"ID":"c139d51d-9d69-4b93-a195-4a8d74dbcabb","Type":"ContainerStarted","Data":"5e20318e8233a54ace88115f370f4bb591b5990bc88a6791e3406b0f15dfb070"} Oct 31 22:09:15.018386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.018116 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:15.050841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.050812 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7"] Oct 31 22:09:15.054335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.054297 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073d4981-pfzf7"] Oct 31 22:09:15.064162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.064120 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" podStartSLOduration=2.064106146 podStartE2EDuration="2.064106146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:15.061024856 +0000 UTC m=+3195.186132110" watchObservedRunningTime="2026-10-31 22:09:15.064106146 +0000 UTC m=+3195.189213379" Oct 31 22:09:15.359315 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.359237 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz"] Oct 31 22:09:15.363678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.363659 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.398928 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.398905 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:09:15.401422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.401403 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz"] Oct 31 22:09:15.406708 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.406690 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:09:15.458638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.458613 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.458746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.458670 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdcl\" (UniqueName: \"kubernetes.io/projected/a8fa7e16-e9db-473b-8338-f80873795d13-kube-api-access-5jdcl\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.458815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.458794 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.559753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.559724 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.559880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.559782 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdcl\" (UniqueName: \"kubernetes.io/projected/a8fa7e16-e9db-473b-8338-f80873795d13-kube-api-access-5jdcl\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.559880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.559848 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.560456 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.560433 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.562163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.562134 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.583576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.583551 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdcl\" (UniqueName: \"kubernetes.io/projected/a8fa7e16-e9db-473b-8338-f80873795d13-kube-api-access-5jdcl\") pod \"maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.678162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.678105 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:15.952884 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.952813 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s"] Oct 31 22:09:15.958199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.958180 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:15.973643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.973621 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-cd421c-dockercfg-79wzx\"" Oct 31 22:09:15.974422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.974403 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-cd421c\"" Oct 31 22:09:15.976825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.976801 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s"] Oct 31 22:09:15.987356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.987324 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb51e128-721e-43ec-a9e5-367d78845641" path="/var/lib/kubelet/pods/fb51e128-721e-43ec-a9e5-367d78845641/volumes" Oct 31 22:09:15.989288 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:15.989268 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz"] Oct 31 22:09:15.991243 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:15.991217 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fa7e16_e9db_473b_8338_f80873795d13.slice/crio-01269448f24e53b7ac2b2112cb13fd59ef9fe07bceb89d58858b426dc052ae6d WatchSource:0}: Error finding container 01269448f24e53b7ac2b2112cb13fd59ef9fe07bceb89d58858b426dc052ae6d: Status 404 returned error can't find the container with id 01269448f24e53b7ac2b2112cb13fd59ef9fe07bceb89d58858b426dc052ae6d Oct 31 22:09:16.183288 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.183254 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhzhj\" (UniqueName: \"kubernetes.io/projected/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-kube-api-access-zhzhj\") pod \"payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:16.183802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.183588 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:16.198878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.196800 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" event={"ID":"a8fa7e16-e9db-473b-8338-f80873795d13","Type":"ContainerStarted","Data":"01269448f24e53b7ac2b2112cb13fd59ef9fe07bceb89d58858b426dc052ae6d"} Oct 31 22:09:16.198878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.197505 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6"] Oct 31 22:09:16.205179 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.205161 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6"] Oct 31 22:09:16.205359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.205292 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.284791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.284755 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5b6cba-bd54-4c8f-a351-26b092e20e20-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.284943 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.284856 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjrv\" (UniqueName: \"kubernetes.io/projected/ca5b6cba-bd54-4c8f-a351-26b092e20e20-kube-api-access-qsjrv\") pod \"payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.285104 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.285082 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:16.285204 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.285191 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhzhj\" (UniqueName: \"kubernetes.io/projected/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-kube-api-access-zhzhj\") pod \"payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:16.285806 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.285789 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:16.308277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.308254 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhzhj\" (UniqueName: \"kubernetes.io/projected/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-kube-api-access-zhzhj\") pod \"payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:16.386460 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.386434 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjrv\" (UniqueName: \"kubernetes.io/projected/ca5b6cba-bd54-4c8f-a351-26b092e20e20-kube-api-access-qsjrv\") pod \"payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.386634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.386621 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5b6cba-bd54-4c8f-a351-26b092e20e20-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.387311 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.387278 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5b6cba-bd54-4c8f-a351-26b092e20e20-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.444110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.444075 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjrv\" (UniqueName: \"kubernetes.io/projected/ca5b6cba-bd54-4c8f-a351-26b092e20e20-kube-api-access-qsjrv\") pod \"payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.525809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.525553 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:16.578793 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.578758 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:16.855648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.855617 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6"] Oct 31 22:09:16.856370 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:16.856340 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5b6cba_bd54_4c8f_a351_26b092e20e20.slice/crio-870499022b5c4c01f8cc6a6daaf55fedb4ee55355715097cc7460e2dca81ab64 WatchSource:0}: Error finding container 870499022b5c4c01f8cc6a6daaf55fedb4ee55355715097cc7460e2dca81ab64: Status 404 returned error can't find the container with id 870499022b5c4c01f8cc6a6daaf55fedb4ee55355715097cc7460e2dca81ab64 Oct 31 22:09:16.911967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:16.911942 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s"] Oct 31 22:09:16.913766 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:16.913743 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c77e2f_514a_4ddf_9f8e_3ec45d1fdbf4.slice/crio-b09b7f4d34f912e890bde2372cce48209e53495b6ffb2782dfeb803847c97456 WatchSource:0}: Error finding container b09b7f4d34f912e890bde2372cce48209e53495b6ffb2782dfeb803847c97456: Status 404 returned error can't find the container with id b09b7f4d34f912e890bde2372cce48209e53495b6ffb2782dfeb803847c97456 Oct 31 22:09:17.204998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.204908 4294 generic.go:356] "Generic (PLEG): container finished" podID="a8fa7e16-e9db-473b-8338-f80873795d13" containerID="f402969b4106ae47c5b0c5328daae159cb5d04485aefc98781b47dfcd0fb0cf1" exitCode=0 Oct 31 22:09:17.205469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.204982 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" event={"ID":"a8fa7e16-e9db-473b-8338-f80873795d13","Type":"ContainerDied","Data":"f402969b4106ae47c5b0c5328daae159cb5d04485aefc98781b47dfcd0fb0cf1"} Oct 31 22:09:17.206760 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.206729 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" event={"ID":"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4","Type":"ContainerStarted","Data":"6ebcbf6ce44f0ba414255b80a4765ba0fe169ced5318e75ecec533d23dbd700f"} Oct 31 22:09:17.206901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.206785 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" event={"ID":"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4","Type":"ContainerStarted","Data":"b09b7f4d34f912e890bde2372cce48209e53495b6ffb2782dfeb803847c97456"} Oct 31 22:09:17.206901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.206810 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:17.208638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.208605 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" event={"ID":"ca5b6cba-bd54-4c8f-a351-26b092e20e20","Type":"ContainerStarted","Data":"86151cacf887b641f5139e4eaaa8690e0a6fc291b1959205f267e212417220d5"} Oct 31 22:09:17.208886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.208663 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" event={"ID":"ca5b6cba-bd54-4c8f-a351-26b092e20e20","Type":"ContainerStarted","Data":"870499022b5c4c01f8cc6a6daaf55fedb4ee55355715097cc7460e2dca81ab64"} Oct 31 22:09:17.208886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.208741 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:17.295699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.295656 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" podStartSLOduration=1.295642558 podStartE2EDuration="1.295642558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:17.29349795 +0000 UTC m=+3197.418605180" watchObservedRunningTime="2026-10-31 22:09:17.295642558 +0000 UTC m=+3197.420749830" Oct 31 22:09:17.335762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:17.335719 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" podStartSLOduration=2.335707941 podStartE2EDuration="2.335707941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:17.334897047 +0000 UTC m=+3197.460004276" watchObservedRunningTime="2026-10-31 22:09:17.335707941 +0000 UTC m=+3197.460815171" Oct 31 22:09:18.096329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.082385 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg"] Oct 31 22:09:18.117198 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.114750 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.117838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.114649 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg"] Oct 31 22:09:18.131676 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.131651 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-943963-openshift-default-dockercfg-ntxzf\"" Oct 31 22:09:18.132223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132189 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-credential-socket\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132371 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132329 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpgk\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-kube-api-access-qrpgk\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132497 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132479 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-socket\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132530 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-token\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132562 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-podinfo\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132591 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-data\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132672 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-certs\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132734 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd00683-6e8f-483e-9d30-93501b4786b8-istiod-ca-cert\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.132794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.132770 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-envoy\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.242953 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.242915 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-certs\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243013 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd00683-6e8f-483e-9d30-93501b4786b8-istiod-ca-cert\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243042 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-envoy\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243100 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-credential-socket\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243160 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpgk\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-kube-api-access-qrpgk\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243254 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-socket\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243283 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-token\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243315 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-podinfo\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243334 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-data\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243706 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-data\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.243993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.243856 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd00683-6e8f-483e-9d30-93501b4786b8-istiod-ca-cert\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.244110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.244095 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-certs\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.244395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.244372 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-socket\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.244766 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.244748 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-credential-socket\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.246960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.246915 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-envoy\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.248864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.248841 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-podinfo\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.268195 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.268173 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-token\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.268837 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.268808 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpgk\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-kube-api-access-qrpgk\") pod \"e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.447191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.447160 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:18.498821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.498779 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:18.547949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.547921 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-xks-ca\") pod \"a8fa7e16-e9db-473b-8338-f80873795d13\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " Oct 31 22:09:18.548132 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.548118 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fa7e16-e9db-473b-8338-f80873795d13-kube-api-access-5jdcl\" (UniqueName: \"kubernetes.io/projected/a8fa7e16-e9db-473b-8338-f80873795d13-kube-api-access-5jdcl\") pod \"a8fa7e16-e9db-473b-8338-f80873795d13\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " Oct 31 22:09:18.548274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.548258 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-openshift-ca\") pod \"a8fa7e16-e9db-473b-8338-f80873795d13\" (UID: \"a8fa7e16-e9db-473b-8338-f80873795d13\") " Oct 31 22:09:18.548656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.548627 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-openshift-ca" pod "a8fa7e16-e9db-473b-8338-f80873795d13" (UID: "a8fa7e16-e9db-473b-8338-f80873795d13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:18.548912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.548888 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:18.550635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.550608 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fa7e16-e9db-473b-8338-f80873795d13-kube-api-access-5jdcl" pod "a8fa7e16-e9db-473b-8338-f80873795d13" (UID: "a8fa7e16-e9db-473b-8338-f80873795d13"). InnerVolumeSpecName "kube-api-access-5jdcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:18.550731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.550706 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-xks-ca" pod "a8fa7e16-e9db-473b-8338-f80873795d13" (UID: "a8fa7e16-e9db-473b-8338-f80873795d13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:09:18.650578 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.650541 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8fa7e16-e9db-473b-8338-f80873795d13-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:18.650578 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.650579 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdcl\" (UniqueName: \"kubernetes.io/projected/a8fa7e16-e9db-473b-8338-f80873795d13-kube-api-access-5jdcl\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:18.840552 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:18.840513 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd00683_6e8f_483e_9d30_93501b4786b8.slice/crio-a7a501435cc9265719dd808a2935e05c3f70303a43e93b748587f7279be3c395 WatchSource:0}: Error finding container a7a501435cc9265719dd808a2935e05c3f70303a43e93b748587f7279be3c395: Status 404 returned error can't find the container with id a7a501435cc9265719dd808a2935e05c3f70303a43e93b748587f7279be3c395 Oct 31 22:09:18.842414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.842389 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg"] Oct 31 22:09:18.843500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.843464 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:09:18.843610 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.843573 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:09:18.843671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:18.843619 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:09:19.255250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.255211 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" event={"ID":"bfd00683-6e8f-483e-9d30-93501b4786b8","Type":"ContainerStarted","Data":"8f9263da6efff226b32bee5b690331ef779113350f9f2fed0c1f4cec422c3aca"} Oct 31 22:09:19.255751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.255281 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" event={"ID":"bfd00683-6e8f-483e-9d30-93501b4786b8","Type":"ContainerStarted","Data":"a7a501435cc9265719dd808a2935e05c3f70303a43e93b748587f7279be3c395"} Oct 31 22:09:19.256943 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.256914 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" Oct 31 22:09:19.257091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.256912 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz" event={"ID":"a8fa7e16-e9db-473b-8338-f80873795d13","Type":"ContainerDied","Data":"01269448f24e53b7ac2b2112cb13fd59ef9fe07bceb89d58858b426dc052ae6d"} Oct 31 22:09:19.257091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.257028 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01269448f24e53b7ac2b2112cb13fd59ef9fe07bceb89d58858b426dc052ae6d" Oct 31 22:09:19.502813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.502260 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:19.502813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.502458 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Oct 31 22:09:19.502813 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.502506 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Oct 31 22:09:19.531379 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:19.530373 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podStartSLOduration=2.530350104 podStartE2EDuration="2.530350104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:19.305516119 +0000 UTC m=+3199.430623350" watchObservedRunningTime="2026-10-31 22:09:19.530350104 +0000 UTC m=+3199.655457336" Oct 31 22:09:20.499949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:20.499910 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Oct 31 22:09:20.500456 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:20.499965 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Oct 31 22:09:21.499429 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:21.499389 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Oct 31 22:09:21.499595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:21.499444 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Oct 31 22:09:22.499681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:22.499644 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Oct 31 22:09:22.500288 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:22.499699 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Oct 31 22:09:23.501792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.501748 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Oct 31 22:09:23.502285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.501808 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Oct 31 22:09:23.681404 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.681355 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx"] Oct 31 22:09:23.682693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.682669 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a8fa7e16-e9db-473b-8338-f80873795d13" containerName="revoke-keys" Oct 31 22:09:23.682862 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.682847 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fa7e16-e9db-473b-8338-f80873795d13" containerName="revoke-keys" Oct 31 22:09:23.683233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.683211 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fa7e16-e9db-473b-8338-f80873795d13" containerName="revoke-keys" Oct 31 22:09:23.700641 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.700553 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:23.706781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.705909 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx"] Oct 31 22:09:23.717540 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.717507 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp"] Oct 31 22:09:23.722972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.722948 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:23.859324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.851853 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rgk\" (UniqueName: \"kubernetes.io/projected/af26ed87-0e43-40ef-9c75-3f6acc1464a4-kube-api-access-x7rgk\") pod \"payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:23.859324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.851963 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af26ed87-0e43-40ef-9c75-3f6acc1464a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:23.879856 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.879824 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c2d1b8-dockercfg-46trp\"" Oct 31 22:09:23.901421 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.890270 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c2d1b8\"" Oct 31 22:09:23.940330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.934348 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp"] Oct 31 22:09:23.957204 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.957170 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rgk\" (UniqueName: \"kubernetes.io/projected/af26ed87-0e43-40ef-9c75-3f6acc1464a4-kube-api-access-x7rgk\") pod \"payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:23.957407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.957259 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af26ed87-0e43-40ef-9c75-3f6acc1464a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:23.957407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.957288 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62125f24-856d-4185-86c0-3e9f46bcb751-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:23.957501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.957415 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ddw\" (UniqueName: \"kubernetes.io/projected/62125f24-856d-4185-86c0-3e9f46bcb751-kube-api-access-m5ddw\") pod \"payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:23.958451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:23.958428 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af26ed87-0e43-40ef-9c75-3f6acc1464a4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:24.004100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.004062 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rgk\" (UniqueName: \"kubernetes.io/projected/af26ed87-0e43-40ef-9c75-3f6acc1464a4-kube-api-access-x7rgk\") pod \"payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:24.059175 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.059103 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62125f24-856d-4185-86c0-3e9f46bcb751-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:24.059438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.059407 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ddw\" (UniqueName: \"kubernetes.io/projected/62125f24-856d-4185-86c0-3e9f46bcb751-kube-api-access-m5ddw\") pod \"payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:24.059897 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.059877 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62125f24-856d-4185-86c0-3e9f46bcb751-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:24.098527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.098493 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ddw\" (UniqueName: \"kubernetes.io/projected/62125f24-856d-4185-86c0-3e9f46bcb751-kube-api-access-m5ddw\") pod \"payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:24.137533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.137510 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:24.202643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.202615 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:24.499977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.499940 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" start-of-body= Oct 31 22:09:24.500127 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.500003 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.35:15021/healthz/ready\": dial tcp 10.134.1.35:15021: connect: connection refused" Oct 31 22:09:24.500699 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:24.500675 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf26ed87_0e43_40ef_9c75_3f6acc1464a4.slice/crio-30481aced7f48ad1ea79f3fada21c81c98f9c579df7ae29a5d721c9c111b1a8a WatchSource:0}: Error finding container 30481aced7f48ad1ea79f3fada21c81c98f9c579df7ae29a5d721c9c111b1a8a: Status 404 returned error can't find the container with id 30481aced7f48ad1ea79f3fada21c81c98f9c579df7ae29a5d721c9c111b1a8a Oct 31 22:09:24.502220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.502188 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx"] Oct 31 22:09:24.600423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:24.600393 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp"] Oct 31 22:09:24.602027 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:24.602004 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62125f24_856d_4185_86c0_3e9f46bcb751.slice/crio-c7bf1a91fd3fb3ba714fd4dca20f0c6c1de9500b71f0ee48b7cec39d47d41cd2 WatchSource:0}: Error finding container c7bf1a91fd3fb3ba714fd4dca20f0c6c1de9500b71f0ee48b7cec39d47d41cd2: Status 404 returned error can't find the container with id c7bf1a91fd3fb3ba714fd4dca20f0c6c1de9500b71f0ee48b7cec39d47d41cd2 Oct 31 22:09:25.357987 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.357947 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" event={"ID":"af26ed87-0e43-40ef-9c75-3f6acc1464a4","Type":"ContainerStarted","Data":"17af1cf45fe28dd9848cfa23d1684fde296fd207e10485e5ae52c709d54deabc"} Oct 31 22:09:25.358183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.358011 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" event={"ID":"af26ed87-0e43-40ef-9c75-3f6acc1464a4","Type":"ContainerStarted","Data":"30481aced7f48ad1ea79f3fada21c81c98f9c579df7ae29a5d721c9c111b1a8a"} Oct 31 22:09:25.358183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.358039 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:25.359518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.359492 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" event={"ID":"62125f24-856d-4185-86c0-3e9f46bcb751","Type":"ContainerStarted","Data":"6778008494f716a0395f9c4c909e026ce6458f54decfabc3790df35986487292"} Oct 31 22:09:25.359632 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.359536 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" event={"ID":"62125f24-856d-4185-86c0-3e9f46bcb751","Type":"ContainerStarted","Data":"c7bf1a91fd3fb3ba714fd4dca20f0c6c1de9500b71f0ee48b7cec39d47d41cd2"} Oct 31 22:09:25.359632 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.359581 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:25.399090 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.399038 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" podStartSLOduration=2.399020399 podStartE2EDuration="2.399020399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:25.398397198 +0000 UTC m=+3205.523504426" watchObservedRunningTime="2026-10-31 22:09:25.399020399 +0000 UTC m=+3205.524127632" Oct 31 22:09:25.440871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.440826 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" podStartSLOduration=2.440816095 podStartE2EDuration="2.440816095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:25.439464137 +0000 UTC m=+3205.564571371" watchObservedRunningTime="2026-10-31 22:09:25.440816095 +0000 UTC m=+3205.565923325" Oct 31 22:09:25.504395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.504369 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:25.504815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.504798 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:25.505412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.505393 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:09:25.703236 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.703151 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn"] Oct 31 22:09:25.730473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:25.730445 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8"] Oct 31 22:09:26.201277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:26.201248 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:09:26.273608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:26.273566 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fc546546-rdlkm"] Oct 31 22:09:28.247340 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:28.247283 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:09:28.247840 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:28.247495 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:09:30.703969 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:30.703926 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" podUID="9035b707-e450-489b-b776-f695d7e1475a" containerName="payload-processing" containerID="cri-o://5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2" gracePeriod=25 Oct 31 22:09:30.730853 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:30.730822 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" podUID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" containerName="payload-pre-processing" containerID="cri-o://c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824" gracePeriod=25 Oct 31 22:09:31.274016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.273976 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" podUID="5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" containerName="payload-processing" containerID="cri-o://93207883830d15726692b4c76bd4d3ee421ffd83a8c5fda8227c59db37e4e8ec" gracePeriod=25 Oct 31 22:09:31.312536 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:09:31.312506 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:09:31.407178 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.407152 4294 generic.go:356] "Generic (PLEG): container finished" podID="5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" containerID="93207883830d15726692b4c76bd4d3ee421ffd83a8c5fda8227c59db37e4e8ec" exitCode=0 Oct 31 22:09:31.407326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.407198 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" event={"ID":"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606","Type":"ContainerDied","Data":"93207883830d15726692b4c76bd4d3ee421ffd83a8c5fda8227c59db37e4e8ec"} Oct 31 22:09:31.542411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.542339 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:09:31.676747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.676707 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-kube-api-access-kmjb5\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-kube-api-access-kmjb5\") pod \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " Oct 31 22:09:31.676940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.676765 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-routing-overlay-volume\") pod \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " Oct 31 22:09:31.676940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.676851 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-provider-credentials\") pod \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " Oct 31 22:09:31.677066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.676939 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-plugins-config-volume\") pod \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\" (UID: \"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606\") " Oct 31 22:09:31.677119 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.677082 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-routing-overlay-volume" pod "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" (UID: "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:31.677321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.677280 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-plugins-config-volume" pod "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" (UID: "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:31.677511 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.677492 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:31.677570 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.677516 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:31.678837 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.678818 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-kube-api-access-kmjb5" pod "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" (UID: "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606"). InnerVolumeSpecName "kube-api-access-kmjb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:31.678936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.678848 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-provider-credentials" pod "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" (UID: "5a35efad-1d5e-4cc8-ac0f-5e65dcb15606"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:31.779283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.779253 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmjb5\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-kube-api-access-kmjb5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:31.779283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:31.779279 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:32.415202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:32.415127 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" Oct 31 22:09:32.415202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:32.415154 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-54fc546546-rdlkm" event={"ID":"5a35efad-1d5e-4cc8-ac0f-5e65dcb15606","Type":"ContainerDied","Data":"ca9283d98673e8f2e9ca3d81e2cf6642ff6da5617ad1c0d84c9f202935973741"} Oct 31 22:09:32.415451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:32.415213 4294 scope.go:122] "RemoveContainer" containerID="93207883830d15726692b4c76bd4d3ee421ffd83a8c5fda8227c59db37e4e8ec" Oct 31 22:09:32.468692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:32.468665 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fc546546-rdlkm"] Oct 31 22:09:32.480387 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:32.480364 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-54fc546546-rdlkm"] Oct 31 22:09:34.024274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:34.024238 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" path="/var/lib/kubelet/pods/5a35efad-1d5e-4cc8-ac0f-5e65dcb15606/volumes" Oct 31 22:09:34.658324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:34.642822 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" podUID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:09:34.662319 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:34.659465 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" podUID="9035b707-e450-489b-b776-f695d7e1475a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:09:34.816606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:34.816560 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz"] Oct 31 22:09:34.842082 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:34.842032 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-fac043-7djjz"] Oct 31 22:09:36.051372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:36.051335 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fa7e16-e9db-473b-8338-f80873795d13" path="/var/lib/kubelet/pods/a8fa7e16-e9db-473b-8338-f80873795d13/volumes" Oct 31 22:09:36.369743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:36.369666 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:09:36.369743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:36.369735 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:09:41.745825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:41.745791 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2"] Oct 31 22:09:41.746454 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:41.746044 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" containerID="cri-o://986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3" gracePeriod=30 Oct 31 22:09:42.364411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.364371 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh"] Oct 31 22:09:42.365081 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.365058 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" containerName="payload-processing" Oct 31 22:09:42.365081 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.365081 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" containerName="payload-processing" Oct 31 22:09:42.365359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.365343 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a35efad-1d5e-4cc8-ac0f-5e65dcb15606" containerName="payload-processing" Oct 31 22:09:42.370112 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.370092 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.384848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.384821 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:09:42.384975 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.384953 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:09:42.386570 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.386552 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh"] Oct 31 22:09:42.440985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.440951 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c716aed1-2695-406b-bf6f-c58422598770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.441350 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.441318 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c716aed1-2695-406b-bf6f-c58422598770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.441478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.441362 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9p9\" (UniqueName: \"kubernetes.io/projected/c716aed1-2695-406b-bf6f-c58422598770-kube-api-access-6v9p9\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.536349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.536323 4294 patch_prober.go:29] interesting pod/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:09:42.536456 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.536372 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:09:42.542966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.542941 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c716aed1-2695-406b-bf6f-c58422598770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.543066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.542971 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9p9\" (UniqueName: \"kubernetes.io/projected/c716aed1-2695-406b-bf6f-c58422598770-kube-api-access-6v9p9\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.543066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.543020 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c716aed1-2695-406b-bf6f-c58422598770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.543654 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.543634 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c716aed1-2695-406b-bf6f-c58422598770-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.545247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.545228 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c716aed1-2695-406b-bf6f-c58422598770-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.566229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.566209 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9p9\" (UniqueName: \"kubernetes.io/projected/c716aed1-2695-406b-bf6f-c58422598770-kube-api-access-6v9p9\") pod \"maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.684809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.684746 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:42.986788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:42.986758 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh"] Oct 31 22:09:42.987096 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:09:42.986961 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc716aed1_2695_406b_bf6f_c58422598770.slice/crio-cc67fdba643316288430d5cc1aa0fbe91e43fa5a5a77c42d5c74cab3fead8f79 WatchSource:0}: Error finding container cc67fdba643316288430d5cc1aa0fbe91e43fa5a5a77c42d5c74cab3fead8f79: Status 404 returned error can't find the container with id cc67fdba643316288430d5cc1aa0fbe91e43fa5a5a77c42d5c74cab3fead8f79 Oct 31 22:09:43.704786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:43.704710 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" event={"ID":"c716aed1-2695-406b-bf6f-c58422598770","Type":"ContainerStarted","Data":"b907e1a8cefdf155866f8f7bead2cb33ceda57d6b90eac115b9e428f85f91687"} Oct 31 22:09:43.704940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:43.704790 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" event={"ID":"c716aed1-2695-406b-bf6f-c58422598770","Type":"ContainerStarted","Data":"cc67fdba643316288430d5cc1aa0fbe91e43fa5a5a77c42d5c74cab3fead8f79"} Oct 31 22:09:43.748998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:43.748956 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" podStartSLOduration=1.7489420610000002 podStartE2EDuration="1.748942061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:43.745226047 +0000 UTC m=+3223.870333277" watchObservedRunningTime="2026-10-31 22:09:43.748942061 +0000 UTC m=+3223.874049291" Oct 31 22:09:44.625105 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:44.625061 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" podUID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:09:44.625502 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:44.625221 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" podUID="9035b707-e450-489b-b776-f695d7e1475a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:09:44.712824 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:44.712794 4294 generic.go:356] "Generic (PLEG): container finished" podID="c716aed1-2695-406b-bf6f-c58422598770" containerID="b907e1a8cefdf155866f8f7bead2cb33ceda57d6b90eac115b9e428f85f91687" exitCode=0 Oct 31 22:09:44.712989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:44.712837 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" event={"ID":"c716aed1-2695-406b-bf6f-c58422598770","Type":"ContainerDied","Data":"b907e1a8cefdf155866f8f7bead2cb33ceda57d6b90eac115b9e428f85f91687"} Oct 31 22:09:45.896416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.896389 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:45.992402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.992367 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c716aed1-2695-406b-bf6f-c58422598770-kube-api-access-6v9p9\" (UniqueName: \"kubernetes.io/projected/c716aed1-2695-406b-bf6f-c58422598770-kube-api-access-6v9p9\") pod \"c716aed1-2695-406b-bf6f-c58422598770\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " Oct 31 22:09:45.992580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.992531 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c716aed1-2695-406b-bf6f-c58422598770-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c716aed1-2695-406b-bf6f-c58422598770-maas-api-openshift-ca\") pod \"c716aed1-2695-406b-bf6f-c58422598770\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " Oct 31 22:09:45.992580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.992572 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c716aed1-2695-406b-bf6f-c58422598770-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c716aed1-2695-406b-bf6f-c58422598770-maas-api-xks-ca\") pod \"c716aed1-2695-406b-bf6f-c58422598770\" (UID: \"c716aed1-2695-406b-bf6f-c58422598770\") " Oct 31 22:09:45.992843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.992817 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c716aed1-2695-406b-bf6f-c58422598770-maas-api-openshift-ca" pod "c716aed1-2695-406b-bf6f-c58422598770" (UID: "c716aed1-2695-406b-bf6f-c58422598770"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:45.993100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.993081 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c716aed1-2695-406b-bf6f-c58422598770-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:45.994672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.994652 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c716aed1-2695-406b-bf6f-c58422598770-kube-api-access-6v9p9" pod "c716aed1-2695-406b-bf6f-c58422598770" (UID: "c716aed1-2695-406b-bf6f-c58422598770"). InnerVolumeSpecName "kube-api-access-6v9p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:45.994672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:45.994660 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c716aed1-2695-406b-bf6f-c58422598770-maas-api-xks-ca" pod "c716aed1-2695-406b-bf6f-c58422598770" (UID: "c716aed1-2695-406b-bf6f-c58422598770"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:09:46.094685 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:46.094665 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c716aed1-2695-406b-bf6f-c58422598770-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:46.094786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:46.094711 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9p9\" (UniqueName: \"kubernetes.io/projected/c716aed1-2695-406b-bf6f-c58422598770-kube-api-access-6v9p9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:46.729809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:46.729776 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" Oct 31 22:09:46.730008 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:46.729808 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh" event={"ID":"c716aed1-2695-406b-bf6f-c58422598770","Type":"ContainerDied","Data":"cc67fdba643316288430d5cc1aa0fbe91e43fa5a5a77c42d5c74cab3fead8f79"} Oct 31 22:09:46.730008 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:46.729882 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc67fdba643316288430d5cc1aa0fbe91e43fa5a5a77c42d5c74cab3fead8f79" Oct 31 22:09:47.026165 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.026142 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 22:09:47.209297 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209267 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-data\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209343 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-kube-api-access-2nt8l\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-kube-api-access-2nt8l\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209382 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-podinfo\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209403 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-envoy\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209434 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-credential-socket\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209513 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-data" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:09:47.209715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209551 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-socket\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209588 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fb4781-a981-49f3-ae57-b638a9dff4bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fb4781-a981-49f3-ae57-b638a9dff4bb-istiod-ca-cert\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209679 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-token\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-token\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.209949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.209747 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-certs\") pod \"82fb4781-a981-49f3-ae57-b638a9dff4bb\" (UID: \"82fb4781-a981-49f3-ae57-b638a9dff4bb\") " Oct 31 22:09:47.210085 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210061 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fb4781-a981-49f3-ae57-b638a9dff4bb-istiod-ca-cert" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:47.210239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210214 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-credential-socket" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:09:47.210330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210252 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-certs" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:09:47.210483 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210460 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-socket" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:09:47.210625 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210607 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.210692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210631 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fb4781-a981-49f3-ae57-b638a9dff4bb-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.210692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210646 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.210692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210660 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.210692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.210672 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.211780 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.211680 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-envoy" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:09:47.211780 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.211752 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-podinfo" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:09:47.211952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.211900 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-kube-api-access-2nt8l" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "kube-api-access-2nt8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:47.212451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.212428 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-token" pod "82fb4781-a981-49f3-ae57-b638a9dff4bb" (UID: "82fb4781-a981-49f3-ae57-b638a9dff4bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:47.311518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.311463 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.311518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.311485 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nt8l\" (UniqueName: \"kubernetes.io/projected/82fb4781-a981-49f3-ae57-b638a9dff4bb-kube-api-access-2nt8l\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.311518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.311496 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.311518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.311504 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fb4781-a981-49f3-ae57-b638a9dff4bb-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:47.738293 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.738261 4294 generic.go:356] "Generic (PLEG): container finished" podID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerID="986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3" exitCode=0 Oct 31 22:09:47.738495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.738332 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" event={"ID":"82fb4781-a981-49f3-ae57-b638a9dff4bb","Type":"ContainerDied","Data":"986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3"} Oct 31 22:09:47.738495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.738396 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" event={"ID":"82fb4781-a981-49f3-ae57-b638a9dff4bb","Type":"ContainerDied","Data":"2ef3e901bd56a7fd76f59d946b41eeac216e396c388642c499b568e2d21514a1"} Oct 31 22:09:47.738495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.738349 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2" Oct 31 22:09:47.738495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.738426 4294 scope.go:122] "RemoveContainer" containerID="986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3" Oct 31 22:09:47.749697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.749677 4294 scope.go:122] "RemoveContainer" containerID="c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a" Oct 31 22:09:47.759095 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.759077 4294 scope.go:122] "RemoveContainer" containerID="986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3" Oct 31 22:09:47.759384 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:09:47.759364 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3\": container with ID starting with 986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3 not found: ID does not exist" containerID="986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3" Oct 31 22:09:47.759454 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.759391 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3"} err="failed to get container status \"986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3\": rpc error: code = NotFound desc = could not find container \"986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3\": container with ID starting with 986250f5e644b6b9841c15b5dd4752065fb2f32877847671ca02cc2d9e6436c3 not found: ID does not exist" Oct 31 22:09:47.759454 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.759410 4294 scope.go:122] "RemoveContainer" containerID="c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a" Oct 31 22:09:47.759639 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:09:47.759621 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a\": container with ID starting with c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a not found: ID does not exist" containerID="c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a" Oct 31 22:09:47.759688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.759647 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a"} err="failed to get container status \"c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a\": rpc error: code = NotFound desc = could not find container \"c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a\": container with ID starting with c1125a230dc3718176088cdde1e6065a2bcf0b8122f6c9dbb5210a385565860a not found: ID does not exist" Oct 31 22:09:47.804857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.804823 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2"] Oct 31 22:09:47.817444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.817416 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-fac043-openshift-default-7bc677c6b9-ld4n2"] Oct 31 22:09:47.990013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:47.989941 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" path="/var/lib/kubelet/pods/82fb4781-a981-49f3-ae57-b638a9dff4bb/volumes" Oct 31 22:09:51.046482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.046459 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 22:09:51.070596 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.070577 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 22:09:51.158279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.158249 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-kube-api-access-bv299\" (UniqueName: \"kubernetes.io/projected/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-kube-api-access-bv299\") pod \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " Oct 31 22:09:51.158466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.158314 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9035b707-e450-489b-b776-f695d7e1475a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9035b707-e450-489b-b776-f695d7e1475a-plugins-config-volume\") pod \"9035b707-e450-489b-b776-f695d7e1475a\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " Oct 31 22:09:51.158466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.158435 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-plugins-config-volume\") pod \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\" (UID: \"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4\") " Oct 31 22:09:51.158686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.158666 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9035b707-e450-489b-b776-f695d7e1475a-kube-api-access-54xpb\" (UniqueName: \"kubernetes.io/projected/9035b707-e450-489b-b776-f695d7e1475a-kube-api-access-54xpb\") pod \"9035b707-e450-489b-b776-f695d7e1475a\" (UID: \"9035b707-e450-489b-b776-f695d7e1475a\") " Oct 31 22:09:51.158792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.158767 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9035b707-e450-489b-b776-f695d7e1475a-plugins-config-volume" pod "9035b707-e450-489b-b776-f695d7e1475a" (UID: "9035b707-e450-489b-b776-f695d7e1475a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:51.158858 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.158810 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-plugins-config-volume" pod "5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" (UID: "5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:09:51.159347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.159325 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:51.159476 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.159351 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9035b707-e450-489b-b776-f695d7e1475a-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:51.160789 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.160766 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-kube-api-access-bv299" pod "5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" (UID: "5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4"). InnerVolumeSpecName "kube-api-access-bv299". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:51.160935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.160909 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9035b707-e450-489b-b776-f695d7e1475a-kube-api-access-54xpb" pod "9035b707-e450-489b-b776-f695d7e1475a" (UID: "9035b707-e450-489b-b776-f695d7e1475a"). InnerVolumeSpecName "kube-api-access-54xpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:09:51.261085 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.261023 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54xpb\" (UniqueName: \"kubernetes.io/projected/9035b707-e450-489b-b776-f695d7e1475a-kube-api-access-54xpb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:51.261085 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.261050 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv299\" (UniqueName: \"kubernetes.io/projected/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4-kube-api-access-bv299\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:09:51.770585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.770553 4294 generic.go:356] "Generic (PLEG): container finished" podID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" containerID="c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824" exitCode=0 Oct 31 22:09:51.770762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.770597 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" Oct 31 22:09:51.770762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.770635 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" event={"ID":"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4","Type":"ContainerDied","Data":"c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824"} Oct 31 22:09:51.770762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.770701 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8" event={"ID":"5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4","Type":"ContainerDied","Data":"2c470970a1e538d519e4595482cef6820d8a8e1afd7ca0385975fa154370f259"} Oct 31 22:09:51.770762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.770732 4294 scope.go:122] "RemoveContainer" containerID="c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824" Oct 31 22:09:51.772372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.772351 4294 generic.go:356] "Generic (PLEG): container finished" podID="9035b707-e450-489b-b776-f695d7e1475a" containerID="5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2" exitCode=0 Oct 31 22:09:51.772512 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.772379 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" event={"ID":"9035b707-e450-489b-b776-f695d7e1475a","Type":"ContainerDied","Data":"5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2"} Oct 31 22:09:51.772512 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.772410 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" Oct 31 22:09:51.772512 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.772413 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn" event={"ID":"9035b707-e450-489b-b776-f695d7e1475a","Type":"ContainerDied","Data":"63d8e34a44c70dc55dfe91bd3f8e7c4715162c7ac594f6edbe7b68a0497785bb"} Oct 31 22:09:51.782158 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.782140 4294 scope.go:122] "RemoveContainer" containerID="c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824" Oct 31 22:09:51.782420 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:09:51.782401 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824\": container with ID starting with c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824 not found: ID does not exist" containerID="c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824" Oct 31 22:09:51.782480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.782425 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824"} err="failed to get container status \"c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824\": rpc error: code = NotFound desc = could not find container \"c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824\": container with ID starting with c52bb6c9c6018548f74a0379385b96b9a3004c459b7f390db5a7b54a0a7c0824 not found: ID does not exist" Oct 31 22:09:51.782480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.782440 4294 scope.go:122] "RemoveContainer" containerID="5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2" Oct 31 22:09:51.791574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.791552 4294 scope.go:122] "RemoveContainer" containerID="5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2" Oct 31 22:09:51.791817 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:09:51.791794 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2\": container with ID starting with 5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2 not found: ID does not exist" containerID="5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2" Oct 31 22:09:51.791871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.791826 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2"} err="failed to get container status \"5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2\": rpc error: code = NotFound desc = could not find container \"5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2\": container with ID starting with 5601a87b789e5bf78d959079672b1cc9582ffce8d6ab67f1fbdb7f1bbb56ddf2 not found: ID does not exist" Oct 31 22:09:51.824555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.824535 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn"] Oct 31 22:09:51.836276 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.836254 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-fac043-7f5bf697-vrxkn"] Oct 31 22:09:51.862986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.862959 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8"] Oct 31 22:09:51.866557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.866534 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fac043-7d54f45fdc-shkg8"] Oct 31 22:09:51.988878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.988833 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" path="/var/lib/kubelet/pods/5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4/volumes" Oct 31 22:09:51.989594 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:51.989568 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9035b707-e450-489b-b776-f695d7e1475a" path="/var/lib/kubelet/pods/9035b707-e450-489b-b776-f695d7e1475a/volumes" Oct 31 22:09:56.302335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302291 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs"] Oct 31 22:09:56.302937 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302921 4294 cpu_manager.go:414] "containerMap: removing container" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" Oct 31 22:09:56.302982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302938 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" Oct 31 22:09:56.302982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302955 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c716aed1-2695-406b-bf6f-c58422598770" containerName="revoke-keys" Oct 31 22:09:56.302982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302961 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c716aed1-2695-406b-bf6f-c58422598770" containerName="revoke-keys" Oct 31 22:09:56.302982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302976 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9035b707-e450-489b-b776-f695d7e1475a" containerName="payload-processing" Oct 31 22:09:56.302982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302980 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9035b707-e450-489b-b776-f695d7e1475a" containerName="payload-processing" Oct 31 22:09:56.303144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.302995 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" containerName="payload-pre-processing" Oct 31 22:09:56.303144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.303000 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" containerName="payload-pre-processing" Oct 31 22:09:56.303144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.303111 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" Oct 31 22:09:56.303144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.303122 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" Oct 31 22:09:56.303144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.303136 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c716aed1-2695-406b-bf6f-c58422598770" containerName="revoke-keys" Oct 31 22:09:56.303144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.303145 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9035b707-e450-489b-b776-f695d7e1475a" containerName="payload-processing" Oct 31 22:09:56.303373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.303152 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e07fcfd-25ad-41a5-abbc-0d9e9c7d29c4" containerName="payload-pre-processing" Oct 31 22:09:56.309060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.309043 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.325192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.325169 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs"] Oct 31 22:09:56.331343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.331322 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:09:56.331515 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.331291 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:09:56.433169 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.433139 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg2jh\" (UniqueName: \"kubernetes.io/projected/0b7390b4-4c4d-4230-b496-374f12ab0736-kube-api-access-rg2jh\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.433354 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.433333 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.433427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.433393 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.534028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.534000 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg2jh\" (UniqueName: \"kubernetes.io/projected/0b7390b4-4c4d-4230-b496-374f12ab0736-kube-api-access-rg2jh\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.534164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.534105 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.534164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.534141 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.534792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.534768 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.536595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.536576 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.559219 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.559160 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg2jh\" (UniqueName: \"kubernetes.io/projected/0b7390b4-4c4d-4230-b496-374f12ab0736-kube-api-access-rg2jh\") pod \"maas-api-revoke-sub-18a39add-gmmcs\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:56.638369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:56.638244 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:09:57.017676 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:57.017641 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs"] Oct 31 22:09:57.827533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:57.827503 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" event={"ID":"0b7390b4-4c4d-4230-b496-374f12ab0736","Type":"ContainerStarted","Data":"a4f4c23b0daf71e2d659b9e5306aaeb52360af943723389c7d41192fee09c54d"} Oct 31 22:09:57.827958 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:57.827550 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" event={"ID":"0b7390b4-4c4d-4230-b496-374f12ab0736","Type":"ContainerStarted","Data":"a7a63ec346d4038f0797917a7f2db4673a640dbc5764198b0e1d99b684c5b621"} Oct 31 22:09:57.873001 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:57.872956 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" podStartSLOduration=1.872942191 podStartE2EDuration="1.872942191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:09:57.870110926 +0000 UTC m=+3237.995218158" watchObservedRunningTime="2026-10-31 22:09:57.872942191 +0000 UTC m=+3237.998049420" Oct 31 22:09:58.841013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:58.840976 4294 generic.go:356] "Generic (PLEG): container finished" podID="0b7390b4-4c4d-4230-b496-374f12ab0736" containerID="a4f4c23b0daf71e2d659b9e5306aaeb52360af943723389c7d41192fee09c54d" exitCode=0 Oct 31 22:09:58.841424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:09:58.841020 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" event={"ID":"0b7390b4-4c4d-4230-b496-374f12ab0736","Type":"ContainerDied","Data":"a4f4c23b0daf71e2d659b9e5306aaeb52360af943723389c7d41192fee09c54d"} Oct 31 22:10:00.028961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.028929 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:10:00.198356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.198241 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-xks-ca\") pod \"0b7390b4-4c4d-4230-b496-374f12ab0736\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " Oct 31 22:10:00.198526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.198362 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7390b4-4c4d-4230-b496-374f12ab0736-kube-api-access-rg2jh\" (UniqueName: \"kubernetes.io/projected/0b7390b4-4c4d-4230-b496-374f12ab0736-kube-api-access-rg2jh\") pod \"0b7390b4-4c4d-4230-b496-374f12ab0736\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " Oct 31 22:10:00.198526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.198392 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-openshift-ca\") pod \"0b7390b4-4c4d-4230-b496-374f12ab0736\" (UID: \"0b7390b4-4c4d-4230-b496-374f12ab0736\") " Oct 31 22:10:00.198816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.198791 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-openshift-ca" pod "0b7390b4-4c4d-4230-b496-374f12ab0736" (UID: "0b7390b4-4c4d-4230-b496-374f12ab0736"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:00.199044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.199022 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:00.200452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.200424 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-xks-ca" pod "0b7390b4-4c4d-4230-b496-374f12ab0736" (UID: "0b7390b4-4c4d-4230-b496-374f12ab0736"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:10:00.200521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.200427 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7390b4-4c4d-4230-b496-374f12ab0736-kube-api-access-rg2jh" pod "0b7390b4-4c4d-4230-b496-374f12ab0736" (UID: "0b7390b4-4c4d-4230-b496-374f12ab0736"). InnerVolumeSpecName "kube-api-access-rg2jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:00.300341 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.300319 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg2jh\" (UniqueName: \"kubernetes.io/projected/0b7390b4-4c4d-4230-b496-374f12ab0736-kube-api-access-rg2jh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:00.300341 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.300342 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7390b4-4c4d-4230-b496-374f12ab0736-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:00.857975 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.857946 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" Oct 31 22:10:00.857975 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.857966 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs" event={"ID":"0b7390b4-4c4d-4230-b496-374f12ab0736","Type":"ContainerDied","Data":"a7a63ec346d4038f0797917a7f2db4673a640dbc5764198b0e1d99b684c5b621"} Oct 31 22:10:00.858254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:00.858035 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a63ec346d4038f0797917a7f2db4673a640dbc5764198b0e1d99b684c5b621" Oct 31 22:10:01.311256 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:10:01.311218 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:10:03.086838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:03.086809 4294 scope.go:122] "RemoveContainer" containerID="e15fac2aed376b85b3c12ce2bfb1cee7fe7efaec51c41730920a0abf94c593ee" Oct 31 22:10:03.098809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:03.098789 4294 scope.go:122] "RemoveContainer" containerID="a77e2f1e31531978eb868cc25cca388f8a12e7b99b86b217e520da57ed47e6f4" Oct 31 22:10:07.465292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:07.465251 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft"] Oct 31 22:10:12.349908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.349875 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx"] Oct 31 22:10:12.350514 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.350500 4294 cpu_manager.go:414] "containerMap: removing container" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" Oct 31 22:10:12.350514 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.350514 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fb4781-a981-49f3-ae57-b638a9dff4bb" containerName="istio-proxy" Oct 31 22:10:12.350612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.350532 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0b7390b4-4c4d-4230-b496-374f12ab0736" containerName="revoke-keys" Oct 31 22:10:12.350612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.350537 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7390b4-4c4d-4230-b496-374f12ab0736" containerName="revoke-keys" Oct 31 22:10:12.350672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.350662 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7390b4-4c4d-4230-b496-374f12ab0736" containerName="revoke-keys" Oct 31 22:10:12.354235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.354207 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.375396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.375367 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx"] Oct 31 22:10:12.383504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.383472 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5affe\"/\"agc-cred-9b790fcbebab\"" Oct 31 22:10:12.383651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.383565 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5affe\"/\"kube-root-ca.crt\"" Oct 31 22:10:12.383651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.383587 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5affe\"/\"routing-overlay\"" Oct 31 22:10:12.383651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.383568 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a5affe\"/\"payload-processing-external-model-e2e--def0cae2-dockercfg-pvvxn\"" Oct 31 22:10:12.383651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.383603 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5affe\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a5affe\"" Oct 31 22:10:12.383651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.383616 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a5affe\"/\"openshift-service-ca.crt\"" Oct 31 22:10:12.457672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.457649 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.457804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.457708 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.457804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.457790 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.457898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.457874 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shwhj\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-kube-api-access-shwhj\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.467367 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.467334 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" podUID="c139d51d-9d69-4b93-a195-4a8d74dbcabb" containerName="payload-processing" containerID="cri-o://f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b" gracePeriod=25 Oct 31 22:10:12.560769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.560346 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.560769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.560459 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.560769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.560501 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.560769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.560602 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shwhj\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-kube-api-access-shwhj\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.562281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.562249 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.563176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.563152 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.581915 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.581887 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.610569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.610462 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shwhj\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-kube-api-access-shwhj\") pod \"payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.668531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.668233 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:12.814840 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.814815 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:10:12.967679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.967606 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-plugins-config-volume\") pod \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " Oct 31 22:10:12.967960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.967699 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-provider-credentials\") pod \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " Oct 31 22:10:12.967960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.967821 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-kube-api-access-mjsdk\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-kube-api-access-mjsdk\") pod \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " Oct 31 22:10:12.967960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.967926 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-routing-overlay-volume\") pod \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\" (UID: \"c139d51d-9d69-4b93-a195-4a8d74dbcabb\") " Oct 31 22:10:12.968134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.968033 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-plugins-config-volume" pod "c139d51d-9d69-4b93-a195-4a8d74dbcabb" (UID: "c139d51d-9d69-4b93-a195-4a8d74dbcabb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:12.968365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.968337 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-routing-overlay-volume" pod "c139d51d-9d69-4b93-a195-4a8d74dbcabb" (UID: "c139d51d-9d69-4b93-a195-4a8d74dbcabb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:12.968601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.968581 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:12.968662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.968611 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c139d51d-9d69-4b93-a195-4a8d74dbcabb-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:12.969868 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.969840 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-provider-credentials" pod "c139d51d-9d69-4b93-a195-4a8d74dbcabb" (UID: "c139d51d-9d69-4b93-a195-4a8d74dbcabb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:12.969986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.969867 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-kube-api-access-mjsdk" pod "c139d51d-9d69-4b93-a195-4a8d74dbcabb" (UID: "c139d51d-9d69-4b93-a195-4a8d74dbcabb"). InnerVolumeSpecName "kube-api-access-mjsdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:12.986931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:12.986907 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx"] Oct 31 22:10:12.987426 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:10:12.987407 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192f0920_4e5e_4727_9eaa_4c93744862fa.slice/crio-6f5c5cd722549b63eefab4c31915a86adc4619e0280b1c74f9d5f93dcffeaca8 WatchSource:0}: Error finding container 6f5c5cd722549b63eefab4c31915a86adc4619e0280b1c74f9d5f93dcffeaca8: Status 404 returned error can't find the container with id 6f5c5cd722549b63eefab4c31915a86adc4619e0280b1c74f9d5f93dcffeaca8 Oct 31 22:10:13.000319 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.000279 4294 generic.go:356] "Generic (PLEG): container finished" podID="c139d51d-9d69-4b93-a195-4a8d74dbcabb" containerID="f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b" exitCode=0 Oct 31 22:10:13.000415 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.000346 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" Oct 31 22:10:13.000415 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.000351 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" event={"ID":"c139d51d-9d69-4b93-a195-4a8d74dbcabb","Type":"ContainerDied","Data":"f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b"} Oct 31 22:10:13.000415 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.000400 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft" event={"ID":"c139d51d-9d69-4b93-a195-4a8d74dbcabb","Type":"ContainerDied","Data":"5e20318e8233a54ace88115f370f4bb591b5990bc88a6791e3406b0f15dfb070"} Oct 31 22:10:13.000574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.000426 4294 scope.go:122] "RemoveContainer" containerID="f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b" Oct 31 22:10:13.001821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.001799 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" event={"ID":"192f0920-4e5e-4727-9eaa-4c93744862fa","Type":"ContainerStarted","Data":"6f5c5cd722549b63eefab4c31915a86adc4619e0280b1c74f9d5f93dcffeaca8"} Oct 31 22:10:13.010633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.010612 4294 scope.go:122] "RemoveContainer" containerID="f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b" Oct 31 22:10:13.010934 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:10:13.010907 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b\": container with ID starting with f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b not found: ID does not exist" containerID="f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b" Oct 31 22:10:13.011007 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.010937 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b"} err="failed to get container status \"f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b\": rpc error: code = NotFound desc = could not find container \"f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b\": container with ID starting with f1e241d85821acb4de40637a9046d9ca36097bba03d4740e3dd64f53021fbd3b not found: ID does not exist" Oct 31 22:10:13.062792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.062763 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft"] Oct 31 22:10:13.069014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.068745 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7f548df5c7-g5nft"] Oct 31 22:10:13.069900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.069875 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:13.070010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.069902 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjsdk\" (UniqueName: \"kubernetes.io/projected/c139d51d-9d69-4b93-a195-4a8d74dbcabb-kube-api-access-mjsdk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:13.367844 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.367811 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww"] Oct 31 22:10:13.368522 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.368506 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c139d51d-9d69-4b93-a195-4a8d74dbcabb" containerName="payload-processing" Oct 31 22:10:13.368522 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.368522 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c139d51d-9d69-4b93-a195-4a8d74dbcabb" containerName="payload-processing" Oct 31 22:10:13.368631 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.368622 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c139d51d-9d69-4b93-a195-4a8d74dbcabb" containerName="payload-processing" Oct 31 22:10:13.372215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.372199 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.388895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.388875 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"payload-processing-external-model-e2e--6e09a196-dockercfg-99t42\"" Oct 31 22:10:13.391196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.391179 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"agc-cred-15a5cdb5c216\"" Oct 31 22:10:13.391334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.391315 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww"] Oct 31 22:10:13.392043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.391980 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"openshift-service-ca.crt\"" Oct 31 22:10:13.398252 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.398232 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"kube-root-ca.crt\"" Oct 31 22:10:13.398345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.398232 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"routing-overlay\"" Oct 31 22:10:13.398345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.398234 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-4275b9\"/\"payload-processing-external-model-plugins-e2e-extmod-a-4275b9\"" Oct 31 22:10:13.475923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.475899 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.476121 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.476107 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.476171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.476137 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.476171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.476157 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7zzv\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-kube-api-access-c7zzv\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.577652 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.577615 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.579161 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.579136 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.579344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.579325 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7zzv\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-kube-api-access-c7zzv\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.579530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.579514 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.580505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.580482 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.592335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.579050 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.592977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.592924 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.618372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.615356 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7zzv\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-kube-api-access-c7zzv\") pod \"payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.686199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.686170 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:13.988503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:13.988469 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c139d51d-9d69-4b93-a195-4a8d74dbcabb" path="/var/lib/kubelet/pods/c139d51d-9d69-4b93-a195-4a8d74dbcabb/volumes" Oct 31 22:10:14.002008 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:14.001982 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww"] Oct 31 22:10:14.002166 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:10:14.001991 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68bab017_a084_448b_b693_a0c9629329f5.slice/crio-64adc76313754cb5e79e42573c1d3adb9316e108a8186368c761e832dfbd6ea7 WatchSource:0}: Error finding container 64adc76313754cb5e79e42573c1d3adb9316e108a8186368c761e832dfbd6ea7: Status 404 returned error can't find the container with id 64adc76313754cb5e79e42573c1d3adb9316e108a8186368c761e832dfbd6ea7 Oct 31 22:10:14.010998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:14.010964 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" event={"ID":"192f0920-4e5e-4727-9eaa-4c93744862fa","Type":"ContainerStarted","Data":"743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e"} Oct 31 22:10:14.011082 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:14.011056 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:14.056554 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:14.056510 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" podStartSLOduration=2.05649804 podStartE2EDuration="2.05649804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:14.052088344 +0000 UTC m=+3254.177195574" watchObservedRunningTime="2026-10-31 22:10:14.05649804 +0000 UTC m=+3254.181605270" Oct 31 22:10:15.019804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:15.019771 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" event={"ID":"68bab017-a084-448b-b693-a0c9629329f5","Type":"ContainerStarted","Data":"2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52"} Oct 31 22:10:15.020284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:15.019821 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" event={"ID":"68bab017-a084-448b-b693-a0c9629329f5","Type":"ContainerStarted","Data":"64adc76313754cb5e79e42573c1d3adb9316e108a8186368c761e832dfbd6ea7"} Oct 31 22:10:15.020284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:15.020051 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:15.064590 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:15.064532 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" podStartSLOduration=2.064514078 podStartE2EDuration="2.064514078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:15.060876106 +0000 UTC m=+3255.185983365" watchObservedRunningTime="2026-10-31 22:10:15.064514078 +0000 UTC m=+3255.189621310" Oct 31 22:10:25.023141 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:25.023104 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:10:26.029019 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:26.028993 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:10:27.358193 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:27.358159 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh"] Oct 31 22:10:27.369663 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:27.369636 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2dbae9-g7pfh"] Oct 31 22:10:27.985642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:27.985613 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c716aed1-2695-406b-bf6f-c58422598770" path="/var/lib/kubelet/pods/c716aed1-2695-406b-bf6f-c58422598770/volumes" Oct 31 22:10:31.310642 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:10:31.310613 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:10:35.267325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:35.263344 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt"] Oct 31 22:10:35.267325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:35.263618 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" containerID="cri-o://2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a" gracePeriod=30 Oct 31 22:10:35.870509 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:35.867427 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2"] Oct 31 22:10:35.944328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:35.918559 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2"] Oct 31 22:10:35.944523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:35.942426 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:35.991368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:35.991340 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:10:35.991652 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:35.991632 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:10:36.050955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.050499 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.050955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.050622 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.050955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.050759 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4bv8\" (UniqueName: \"kubernetes.io/projected/994e4e92-f1c1-4f8c-8440-bfe426a248ed-kube-api-access-g4bv8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.152222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.152137 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4bv8\" (UniqueName: \"kubernetes.io/projected/994e4e92-f1c1-4f8c-8440-bfe426a248ed-kube-api-access-g4bv8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.152790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.152764 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.154920 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.154899 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.155615 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.154386 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.159458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.157782 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.182993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.182945 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4bv8\" (UniqueName: \"kubernetes.io/projected/994e4e92-f1c1-4f8c-8440-bfe426a248ed-kube-api-access-g4bv8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.263184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.262957 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:36.719747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:36.719716 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2"] Oct 31 22:10:36.720996 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:10:36.720971 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994e4e92_f1c1_4f8c_8440_bfe426a248ed.slice/crio-c142b56556e909b3e70b961dbb2c1cb3dfa82e72bd1e9396117cfe764a3b6860 WatchSource:0}: Error finding container c142b56556e909b3e70b961dbb2c1cb3dfa82e72bd1e9396117cfe764a3b6860: Status 404 returned error can't find the container with id c142b56556e909b3e70b961dbb2c1cb3dfa82e72bd1e9396117cfe764a3b6860 Oct 31 22:10:37.409128 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:37.409083 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" event={"ID":"994e4e92-f1c1-4f8c-8440-bfe426a248ed","Type":"ContainerStarted","Data":"882e643115ddf8b7aac5e4ea4b8dc50b9d7ae25aeb180a1f8cadb7fd7df80c5d"} Oct 31 22:10:37.409370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:37.409183 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" event={"ID":"994e4e92-f1c1-4f8c-8440-bfe426a248ed","Type":"ContainerStarted","Data":"c142b56556e909b3e70b961dbb2c1cb3dfa82e72bd1e9396117cfe764a3b6860"} Oct 31 22:10:37.456648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:37.456578 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" podStartSLOduration=2.456560177 podStartE2EDuration="2.456560177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:37.450972888 +0000 UTC m=+3277.576080118" watchObservedRunningTime="2026-10-31 22:10:37.456560177 +0000 UTC m=+3277.581667408" Oct 31 22:10:37.561445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:37.561407 4294 patch_prober.go:29] interesting pod/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:10:37.561642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:37.561471 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:10:38.418430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:38.418393 4294 generic.go:356] "Generic (PLEG): container finished" podID="994e4e92-f1c1-4f8c-8440-bfe426a248ed" containerID="882e643115ddf8b7aac5e4ea4b8dc50b9d7ae25aeb180a1f8cadb7fd7df80c5d" exitCode=0 Oct 31 22:10:38.418844 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:38.418475 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" event={"ID":"994e4e92-f1c1-4f8c-8440-bfe426a248ed","Type":"ContainerDied","Data":"882e643115ddf8b7aac5e4ea4b8dc50b9d7ae25aeb180a1f8cadb7fd7df80c5d"} Oct 31 22:10:39.583902 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.583871 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:39.619567 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.619529 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-xks-ca\") pod \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " Oct 31 22:10:39.619763 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.619743 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-openshift-ca\") pod \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " Oct 31 22:10:39.619846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.619831 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994e4e92-f1c1-4f8c-8440-bfe426a248ed-kube-api-access-g4bv8\" (UniqueName: \"kubernetes.io/projected/994e4e92-f1c1-4f8c-8440-bfe426a248ed-kube-api-access-g4bv8\") pod \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\" (UID: \"994e4e92-f1c1-4f8c-8440-bfe426a248ed\") " Oct 31 22:10:39.620156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.620125 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-openshift-ca" pod "994e4e92-f1c1-4f8c-8440-bfe426a248ed" (UID: "994e4e92-f1c1-4f8c-8440-bfe426a248ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:39.620448 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.620430 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:39.621653 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.621636 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-xks-ca" pod "994e4e92-f1c1-4f8c-8440-bfe426a248ed" (UID: "994e4e92-f1c1-4f8c-8440-bfe426a248ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:10:39.622121 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.622107 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994e4e92-f1c1-4f8c-8440-bfe426a248ed-kube-api-access-g4bv8" pod "994e4e92-f1c1-4f8c-8440-bfe426a248ed" (UID: "994e4e92-f1c1-4f8c-8440-bfe426a248ed"). InnerVolumeSpecName "kube-api-access-g4bv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:39.722472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.722396 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994e4e92-f1c1-4f8c-8440-bfe426a248ed-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:39.722472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:39.722426 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4bv8\" (UniqueName: \"kubernetes.io/projected/994e4e92-f1c1-4f8c-8440-bfe426a248ed-kube-api-access-g4bv8\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.350942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.350864 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt"] Oct 31 22:10:40.351528 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.351513 4294 cpu_manager.go:414] "containerMap: removing container" podUID="994e4e92-f1c1-4f8c-8440-bfe426a248ed" containerName="revoke-keys" Oct 31 22:10:40.351572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.351529 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994e4e92-f1c1-4f8c-8440-bfe426a248ed" containerName="revoke-keys" Oct 31 22:10:40.351666 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.351657 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="994e4e92-f1c1-4f8c-8440-bfe426a248ed" containerName="revoke-keys" Oct 31 22:10:40.355364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.355345 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.372634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.372606 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt"] Oct 31 22:10:40.434185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.434152 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b50a00-9932-429c-933f-97752e1e4923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.435593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.435561 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b50a00-9932-429c-933f-97752e1e4923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.435848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.435831 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rp9\" (UniqueName: \"kubernetes.io/projected/03b50a00-9932-429c-933f-97752e1e4923-kube-api-access-c2rp9\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.440866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.440833 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" event={"ID":"994e4e92-f1c1-4f8c-8440-bfe426a248ed","Type":"ContainerDied","Data":"c142b56556e909b3e70b961dbb2c1cb3dfa82e72bd1e9396117cfe764a3b6860"} Oct 31 22:10:40.441227 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.441193 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c142b56556e909b3e70b961dbb2c1cb3dfa82e72bd1e9396117cfe764a3b6860" Oct 31 22:10:40.441227 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.441141 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2" Oct 31 22:10:40.537214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.537186 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b50a00-9932-429c-933f-97752e1e4923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.537451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.537434 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b50a00-9932-429c-933f-97752e1e4923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.537504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.537473 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rp9\" (UniqueName: \"kubernetes.io/projected/03b50a00-9932-429c-933f-97752e1e4923-kube-api-access-c2rp9\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.538043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.538021 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b50a00-9932-429c-933f-97752e1e4923-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.539492 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.539470 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b50a00-9932-429c-933f-97752e1e4923-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.560396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.560376 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rp9\" (UniqueName: \"kubernetes.io/projected/03b50a00-9932-429c-933f-97752e1e4923-kube-api-access-c2rp9\") pod \"maas-api-revoke-sub-e7a60dc9-lsvwt\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.670013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.669769 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:40.800032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.800007 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 22:10:40.843688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.843379 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istiod-ca-cert\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.844608 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-data\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.844658 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-certs\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.844760 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-token\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-token\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.844801 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-socket\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.844371 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istiod-ca-cert" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.845342 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-data" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.845392 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-podinfo\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.845540 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-socket" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.845756 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-envoy\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.845844 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-credential-socket\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.844903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.845907 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-kube-api-access-b5w88\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-kube-api-access-b5w88\") pod \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\" (UID: \"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573\") " Oct 31 22:10:40.848482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.847976 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.848482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.847997 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.848482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.848010 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.850439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.849412 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-certs" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:10:40.850439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.850371 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-credential-socket" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:10:40.853039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.853017 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-kube-api-access-b5w88" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "kube-api-access-b5w88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:40.853146 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.853052 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-token" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:40.853812 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.853784 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-envoy" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:10:40.854285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.854263 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-podinfo" pod "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" (UID: "06b82ddc-e4ee-40fc-ab6b-e45b30ba9573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:10:40.948914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.948849 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.948914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.948876 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.948914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.948887 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5w88\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-kube-api-access-b5w88\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.948914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.948896 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.948914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.948906 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.948914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.948914 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:40.984105 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:10:40.984078 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b50a00_9932_429c_933f_97752e1e4923.slice/crio-9af07a21e13ffcc56231f85554f6fb44f8483754cee4c500a4bb40ef1db99fa5 WatchSource:0}: Error finding container 9af07a21e13ffcc56231f85554f6fb44f8483754cee4c500a4bb40ef1db99fa5: Status 404 returned error can't find the container with id 9af07a21e13ffcc56231f85554f6fb44f8483754cee4c500a4bb40ef1db99fa5 Oct 31 22:10:40.984219 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:40.984100 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt"] Oct 31 22:10:41.452343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.452293 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" event={"ID":"03b50a00-9932-429c-933f-97752e1e4923","Type":"ContainerStarted","Data":"bb1f4209bd5fd847092a80e75d9db7e826272545ee8510aa7c4099dd8e09a4dd"} Oct 31 22:10:41.452536 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.452375 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" event={"ID":"03b50a00-9932-429c-933f-97752e1e4923","Type":"ContainerStarted","Data":"9af07a21e13ffcc56231f85554f6fb44f8483754cee4c500a4bb40ef1db99fa5"} Oct 31 22:10:41.454131 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.454103 4294 generic.go:356] "Generic (PLEG): container finished" podID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerID="2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a" exitCode=0 Oct 31 22:10:41.454259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.454164 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" Oct 31 22:10:41.454259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.454206 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" event={"ID":"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573","Type":"ContainerDied","Data":"2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a"} Oct 31 22:10:41.454353 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.454261 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt" event={"ID":"06b82ddc-e4ee-40fc-ab6b-e45b30ba9573","Type":"ContainerDied","Data":"1aca8571888123472b29515a80ade6c9f7cdaf82041b5116c45b171e7b00e24e"} Oct 31 22:10:41.454353 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.454290 4294 scope.go:122] "RemoveContainer" containerID="2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a" Oct 31 22:10:41.465136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.465120 4294 scope.go:122] "RemoveContainer" containerID="4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b" Oct 31 22:10:41.475930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.475914 4294 scope.go:122] "RemoveContainer" containerID="2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a" Oct 31 22:10:41.476230 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:10:41.476210 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a\": container with ID starting with 2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a not found: ID does not exist" containerID="2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a" Oct 31 22:10:41.476286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.476237 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a"} err="failed to get container status \"2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a\": rpc error: code = NotFound desc = could not find container \"2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a\": container with ID starting with 2c99985af7335bb05bd5b99aa2498d323e978d0b01f6b1553db0412b69c82d4a not found: ID does not exist" Oct 31 22:10:41.476286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.476255 4294 scope.go:122] "RemoveContainer" containerID="4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b" Oct 31 22:10:41.476528 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:10:41.476513 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b\": container with ID starting with 4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b not found: ID does not exist" containerID="4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b" Oct 31 22:10:41.476586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.476531 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b"} err="failed to get container status \"4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b\": rpc error: code = NotFound desc = could not find container \"4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b\": container with ID starting with 4819dcab46b8bc1ac2d7a837141cde50e7017cb956cd92628bdb49a6f4e8157b not found: ID does not exist" Oct 31 22:10:41.493845 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.493807 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" podStartSLOduration=1.493796874 podStartE2EDuration="1.493796874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:41.492630059 +0000 UTC m=+3281.617737306" watchObservedRunningTime="2026-10-31 22:10:41.493796874 +0000 UTC m=+3281.618904104" Oct 31 22:10:41.531362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.531313 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt"] Oct 31 22:10:41.543381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.543355 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2dbae9-openshift-default-7d88757d4f-strxt"] Oct 31 22:10:41.987126 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:41.987093 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" path="/var/lib/kubelet/pods/06b82ddc-e4ee-40fc-ab6b-e45b30ba9573/volumes" Oct 31 22:10:42.463549 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:42.463516 4294 generic.go:356] "Generic (PLEG): container finished" podID="03b50a00-9932-429c-933f-97752e1e4923" containerID="bb1f4209bd5fd847092a80e75d9db7e826272545ee8510aa7c4099dd8e09a4dd" exitCode=0 Oct 31 22:10:42.463727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:42.463605 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" event={"ID":"03b50a00-9932-429c-933f-97752e1e4923","Type":"ContainerDied","Data":"bb1f4209bd5fd847092a80e75d9db7e826272545ee8510aa7c4099dd8e09a4dd"} Oct 31 22:10:43.630605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.630582 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:43.686609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.686578 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03b50a00-9932-429c-933f-97752e1e4923-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b50a00-9932-429c-933f-97752e1e4923-maas-api-xks-ca\") pod \"03b50a00-9932-429c-933f-97752e1e4923\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " Oct 31 22:10:43.686792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.686756 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b50a00-9932-429c-933f-97752e1e4923-kube-api-access-c2rp9\" (UniqueName: \"kubernetes.io/projected/03b50a00-9932-429c-933f-97752e1e4923-kube-api-access-c2rp9\") pod \"03b50a00-9932-429c-933f-97752e1e4923\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " Oct 31 22:10:43.686864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.686835 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b50a00-9932-429c-933f-97752e1e4923-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b50a00-9932-429c-933f-97752e1e4923-maas-api-openshift-ca\") pod \"03b50a00-9932-429c-933f-97752e1e4923\" (UID: \"03b50a00-9932-429c-933f-97752e1e4923\") " Oct 31 22:10:43.687397 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.687371 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b50a00-9932-429c-933f-97752e1e4923-maas-api-openshift-ca" pod "03b50a00-9932-429c-933f-97752e1e4923" (UID: "03b50a00-9932-429c-933f-97752e1e4923"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:43.688836 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.688807 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03b50a00-9932-429c-933f-97752e1e4923-maas-api-xks-ca" pod "03b50a00-9932-429c-933f-97752e1e4923" (UID: "03b50a00-9932-429c-933f-97752e1e4923"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:10:43.689548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.689514 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b50a00-9932-429c-933f-97752e1e4923-kube-api-access-c2rp9" pod "03b50a00-9932-429c-933f-97752e1e4923" (UID: "03b50a00-9932-429c-933f-97752e1e4923"). InnerVolumeSpecName "kube-api-access-c2rp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:43.788834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.788810 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2rp9\" (UniqueName: \"kubernetes.io/projected/03b50a00-9932-429c-933f-97752e1e4923-kube-api-access-c2rp9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:43.788834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.788835 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03b50a00-9932-429c-933f-97752e1e4923-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:43.789009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:43.788846 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03b50a00-9932-429c-933f-97752e1e4923-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:44.481086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:44.481052 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" Oct 31 22:10:44.481644 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:44.481053 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt" event={"ID":"03b50a00-9932-429c-933f-97752e1e4923","Type":"ContainerDied","Data":"9af07a21e13ffcc56231f85554f6fb44f8483754cee4c500a4bb40ef1db99fa5"} Oct 31 22:10:44.481644 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:44.481196 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9af07a21e13ffcc56231f85554f6fb44f8483754cee4c500a4bb40ef1db99fa5" Oct 31 22:10:46.043045 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.042987 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z"] Oct 31 22:10:46.044564 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.044539 4294 cpu_manager.go:414] "containerMap: removing container" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" Oct 31 22:10:46.044713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.044700 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" Oct 31 22:10:46.044864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.044854 4294 cpu_manager.go:414] "containerMap: removing container" podUID="03b50a00-9932-429c-933f-97752e1e4923" containerName="revoke-keys" Oct 31 22:10:46.044970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.044937 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b50a00-9932-429c-933f-97752e1e4923" containerName="revoke-keys" Oct 31 22:10:46.045118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.045105 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b50a00-9932-429c-933f-97752e1e4923" containerName="revoke-keys" Oct 31 22:10:46.045188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.045122 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" Oct 31 22:10:46.045188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.045136 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" Oct 31 22:10:46.049518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.049496 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.069700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.069670 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z"] Oct 31 22:10:46.078365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.074964 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c08360-openshift-default-dockercfg-j52vj\"" Oct 31 22:10:46.118386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.118344 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-socket\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.118558 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.118421 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frhgb\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-kube-api-access-frhgb\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.118606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.118578 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-credential-socket\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.118704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.118684 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-podinfo\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.118758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.118741 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-token\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.118919 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.118892 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-data\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.119049 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.119027 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istiod-ca-cert\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.119122 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.119066 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-envoy\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.119122 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.119095 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-certs\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.221622 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.221580 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-credential-socket\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.221830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.221675 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-podinfo\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.221830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.221725 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-token\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.221830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.221811 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-data\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222008 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istiod-ca-cert\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222104 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222064 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-envoy\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222104 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222096 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-certs\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222157 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-credential-socket\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222159 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-socket\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222211 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-data\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222246 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frhgb\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-kube-api-access-frhgb\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222564 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-socket\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.222864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.222841 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-certs\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.223187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.223167 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istiod-ca-cert\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.234331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.232977 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-podinfo\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.234331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.233264 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-envoy\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.256820 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.256745 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-token\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.257636 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.257609 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frhgb\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-kube-api-access-frhgb\") pod \"e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.375524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.375437 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:46.793841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.793803 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z"] Oct 31 22:10:46.794835 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:10:46.794803 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1c20cf_b63c_4934_b0e9_485b2df551f4.slice/crio-107f07a0fae215e5ddfc71965d6ed630820a621f4a5d6e5fe3aead48028ce6c3 WatchSource:0}: Error finding container 107f07a0fae215e5ddfc71965d6ed630820a621f4a5d6e5fe3aead48028ce6c3: Status 404 returned error can't find the container with id 107f07a0fae215e5ddfc71965d6ed630820a621f4a5d6e5fe3aead48028ce6c3 Oct 31 22:10:46.798609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.798563 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:10:46.798726 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.798655 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:10:46.798790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:46.798721 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:10:47.511327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:47.511274 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" event={"ID":"6a1c20cf-b63c-4934-b0e9-485b2df551f4","Type":"ContainerStarted","Data":"f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347"} Oct 31 22:10:47.511829 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:47.511353 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" event={"ID":"6a1c20cf-b63c-4934-b0e9-485b2df551f4","Type":"ContainerStarted","Data":"107f07a0fae215e5ddfc71965d6ed630820a621f4a5d6e5fe3aead48028ce6c3"} Oct 31 22:10:47.576042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:47.575988 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podStartSLOduration=1.575973488 podStartE2EDuration="1.575973488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:47.572593804 +0000 UTC m=+3287.697701046" watchObservedRunningTime="2026-10-31 22:10:47.575973488 +0000 UTC m=+3287.701080719" Oct 31 22:10:48.376032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:48.375998 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:48.382707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:48.382683 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:48.519422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:48.519394 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:48.520434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:48.520415 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:10:51.239878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.239844 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt"] Oct 31 22:10:51.240522 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.240507 4294 cpu_manager.go:414] "containerMap: removing container" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" Oct 31 22:10:51.240581 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.240522 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06b82ddc-e4ee-40fc-ab6b-e45b30ba9573" containerName="istio-proxy" Oct 31 22:10:51.244026 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.244009 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.262234 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.262209 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt"] Oct 31 22:10:51.268725 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.268704 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:10:51.268849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.268749 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:10:51.420459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.420429 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qdxx\" (UniqueName: \"kubernetes.io/projected/a0f47986-a737-458f-92a2-dcfa08165bc1-kube-api-access-4qdxx\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.420646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.420629 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.420718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.420690 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.521690 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.521608 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.521828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.521708 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qdxx\" (UniqueName: \"kubernetes.io/projected/a0f47986-a737-458f-92a2-dcfa08165bc1-kube-api-access-4qdxx\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.521952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.521934 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.522521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.522503 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.523919 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.523900 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.545191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.545169 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qdxx\" (UniqueName: \"kubernetes.io/projected/a0f47986-a737-458f-92a2-dcfa08165bc1-kube-api-access-4qdxx\") pod \"maas-api-revoke-sub-5be66063-qwmxt\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.562051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.562027 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:51.875125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:51.875089 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt"] Oct 31 22:10:51.875326 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:10:51.875288 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f47986_a737_458f_92a2_dcfa08165bc1.slice/crio-fbac5f674aa7867658c290bb1a3ce807f33ab4e50328cdc4d6772192331855c9 WatchSource:0}: Error finding container fbac5f674aa7867658c290bb1a3ce807f33ab4e50328cdc4d6772192331855c9: Status 404 returned error can't find the container with id fbac5f674aa7867658c290bb1a3ce807f33ab4e50328cdc4d6772192331855c9 Oct 31 22:10:52.548372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:52.548337 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" event={"ID":"a0f47986-a737-458f-92a2-dcfa08165bc1","Type":"ContainerStarted","Data":"163f5f9f21c25a219e5341c942bbf4b6664c9fd6982117918d18f2d8adc26a37"} Oct 31 22:10:52.548777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:52.548396 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" event={"ID":"a0f47986-a737-458f-92a2-dcfa08165bc1","Type":"ContainerStarted","Data":"fbac5f674aa7867658c290bb1a3ce807f33ab4e50328cdc4d6772192331855c9"} Oct 31 22:10:52.594557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:52.594507 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" podStartSLOduration=1.5944920489999999 podStartE2EDuration="1.594492049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:52.591988281 +0000 UTC m=+3292.717095511" watchObservedRunningTime="2026-10-31 22:10:52.594492049 +0000 UTC m=+3292.719599280" Oct 31 22:10:53.566406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:53.566355 4294 generic.go:356] "Generic (PLEG): container finished" podID="a0f47986-a737-458f-92a2-dcfa08165bc1" containerID="163f5f9f21c25a219e5341c942bbf4b6664c9fd6982117918d18f2d8adc26a37" exitCode=0 Oct 31 22:10:53.566831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:53.566426 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" event={"ID":"a0f47986-a737-458f-92a2-dcfa08165bc1","Type":"ContainerDied","Data":"163f5f9f21c25a219e5341c942bbf4b6664c9fd6982117918d18f2d8adc26a37"} Oct 31 22:10:54.727039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.727015 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:54.798892 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.798857 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-openshift-ca\") pod \"a0f47986-a737-458f-92a2-dcfa08165bc1\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " Oct 31 22:10:54.799079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.798956 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-xks-ca\") pod \"a0f47986-a737-458f-92a2-dcfa08165bc1\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " Oct 31 22:10:54.799166 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.799134 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f47986-a737-458f-92a2-dcfa08165bc1-kube-api-access-4qdxx\" (UniqueName: \"kubernetes.io/projected/a0f47986-a737-458f-92a2-dcfa08165bc1-kube-api-access-4qdxx\") pod \"a0f47986-a737-458f-92a2-dcfa08165bc1\" (UID: \"a0f47986-a737-458f-92a2-dcfa08165bc1\") " Oct 31 22:10:54.799312 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.799274 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-openshift-ca" pod "a0f47986-a737-458f-92a2-dcfa08165bc1" (UID: "a0f47986-a737-458f-92a2-dcfa08165bc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:10:54.799815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.799793 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:54.802134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.802105 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-xks-ca" pod "a0f47986-a737-458f-92a2-dcfa08165bc1" (UID: "a0f47986-a737-458f-92a2-dcfa08165bc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:10:54.812487 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.812467 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f47986-a737-458f-92a2-dcfa08165bc1-kube-api-access-4qdxx" pod "a0f47986-a737-458f-92a2-dcfa08165bc1" (UID: "a0f47986-a737-458f-92a2-dcfa08165bc1"). InnerVolumeSpecName "kube-api-access-4qdxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:10:54.901533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.901450 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qdxx\" (UniqueName: \"kubernetes.io/projected/a0f47986-a737-458f-92a2-dcfa08165bc1-kube-api-access-4qdxx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:54.901533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:54.901478 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f47986-a737-458f-92a2-dcfa08165bc1-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:10:55.430113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.430079 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp"] Oct 31 22:10:55.430822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.430794 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a0f47986-a737-458f-92a2-dcfa08165bc1" containerName="revoke-keys" Oct 31 22:10:55.430822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.430822 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f47986-a737-458f-92a2-dcfa08165bc1" containerName="revoke-keys" Oct 31 22:10:55.431482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.430970 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f47986-a737-458f-92a2-dcfa08165bc1" containerName="revoke-keys" Oct 31 22:10:55.434845 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.434826 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.452639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.452610 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp"] Oct 31 22:10:55.458697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.458673 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-710927\"/\"openshift-service-ca.crt\"" Oct 31 22:10:55.459390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.458766 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-710927\"/\"kube-root-ca.crt\"" Oct 31 22:10:55.461750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.461725 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-710927\"/\"default-dockercfg-8q6wj\"" Oct 31 22:10:55.461848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.461727 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-710927\"/\"test-model-710927-kserve-self-signed-certs\"" Oct 31 22:10:55.508728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.508700 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-model-cache\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.508915 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.508895 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tls-certs\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.509034 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.508954 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tmp-dir\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.509233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.509216 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-dshm\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.509343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.509328 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-home\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.509421 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.509404 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx2g6\" (UniqueName: \"kubernetes.io/projected/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-kube-api-access-rx2g6\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.586679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.586648 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" Oct 31 22:10:55.587286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.586680 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt" event={"ID":"a0f47986-a737-458f-92a2-dcfa08165bc1","Type":"ContainerDied","Data":"fbac5f674aa7867658c290bb1a3ce807f33ab4e50328cdc4d6772192331855c9"} Oct 31 22:10:55.587286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.586739 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbac5f674aa7867658c290bb1a3ce807f33ab4e50328cdc4d6772192331855c9" Oct 31 22:10:55.610901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.610868 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-home\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.610940 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx2g6\" (UniqueName: \"kubernetes.io/projected/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-kube-api-access-rx2g6\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.610976 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-model-cache\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.611001 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tls-certs\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611245 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.611081 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tmp-dir\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.611336 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-home\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.611346 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-dshm\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.611426 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-model-cache\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.611619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.611596 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tmp-dir\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.614039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.614009 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-dshm\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.614417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.614396 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tls-certs\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.634889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.634864 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx2g6\" (UniqueName: \"kubernetes.io/projected/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-kube-api-access-rx2g6\") pod \"test-model-710927-kserve-574cf84d79-kwpvp\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:55.756643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:55.756606 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:10:56.107693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:56.107663 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp"] Oct 31 22:10:56.107898 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:10:56.107875 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9bb315b_5ef8_4c5c_8eb3_79ea3e14b405.slice/crio-4fb2d110053608f34b44b033a3b5b4696530a0833de7a5dc7854c37421e82d43 WatchSource:0}: Error finding container 4fb2d110053608f34b44b033a3b5b4696530a0833de7a5dc7854c37421e82d43: Status 404 returned error can't find the container with id 4fb2d110053608f34b44b033a3b5b4696530a0833de7a5dc7854c37421e82d43 Oct 31 22:10:56.595685 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:56.595643 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" event={"ID":"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405","Type":"ContainerStarted","Data":"e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb"} Oct 31 22:10:56.595877 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:56.595694 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" event={"ID":"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405","Type":"ContainerStarted","Data":"4fb2d110053608f34b44b033a3b5b4696530a0833de7a5dc7854c37421e82d43"} Oct 31 22:10:56.638415 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:56.638371 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" podStartSLOduration=1.638357814 podStartE2EDuration="1.638357814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:10:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:10:56.6372913 +0000 UTC m=+3296.762398529" watchObservedRunningTime="2026-10-31 22:10:56.638357814 +0000 UTC m=+3296.763465061" Oct 31 22:10:56.763883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:10:56.761462 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:11:00.599138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:00.598869 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:11:00.619637 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:00.619605 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:11:01.311357 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:01.311315 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:11:03.157325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.155515 4294 scope.go:122] "RemoveContainer" containerID="60733d7bee8c0397bc48b9859b852bc756da4213cb2d913f879b697a11ea1046" Oct 31 22:11:03.318770 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.318733 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9"] Oct 31 22:11:03.324580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.324555 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.342764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.342724 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9"] Oct 31 22:11:03.435044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.434970 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.435044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.435007 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.435337 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.435282 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wkwd\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-kube-api-access-9wkwd\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.435418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.435402 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.537262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.537225 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.537262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.537266 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.537566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.537531 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wkwd\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-kube-api-access-9wkwd\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.537639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.537611 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.538193 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.538163 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.538368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.538346 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.540010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.539986 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.561683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.561647 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wkwd\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-kube-api-access-9wkwd\") pod \"payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:03.584375 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.584334 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl"] Oct 31 22:11:03.584549 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.584529 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" podUID="a1e93389-0027-4e47-adb2-176eef3e5880" containerName="sim" containerID="cri-o://fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787" gracePeriod=30 Oct 31 22:11:03.638987 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:03.638958 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:04.080293 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.080268 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:11:04.101957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.101900 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e93389-0027-4e47-adb2-176eef3e5880-kube-api-access-jtbbf\" (UniqueName: \"kubernetes.io/projected/a1e93389-0027-4e47-adb2-176eef3e5880-kube-api-access-jtbbf\") pod \"a1e93389-0027-4e47-adb2-176eef3e5880\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " Oct 31 22:11:04.102127 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.102073 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1e93389-0027-4e47-adb2-176eef3e5880-tls\" (UniqueName: \"kubernetes.io/secret/a1e93389-0027-4e47-adb2-176eef3e5880-tls\") pod \"a1e93389-0027-4e47-adb2-176eef3e5880\" (UID: \"a1e93389-0027-4e47-adb2-176eef3e5880\") " Oct 31 22:11:04.104818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.104787 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e93389-0027-4e47-adb2-176eef3e5880-kube-api-access-jtbbf" pod "a1e93389-0027-4e47-adb2-176eef3e5880" (UID: "a1e93389-0027-4e47-adb2-176eef3e5880"). InnerVolumeSpecName "kube-api-access-jtbbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:04.104935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.104813 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1e93389-0027-4e47-adb2-176eef3e5880-tls" pod "a1e93389-0027-4e47-adb2-176eef3e5880" (UID: "a1e93389-0027-4e47-adb2-176eef3e5880"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:11:04.153108 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.153076 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9"] Oct 31 22:11:04.155104 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:04.155076 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afad580_b8d1_4293_8b21_176a395c3f4d.slice/crio-adf151a9f5eeefe67cd2a14ad01f4eae7e43949325386efd86e050702792e35d WatchSource:0}: Error finding container adf151a9f5eeefe67cd2a14ad01f4eae7e43949325386efd86e050702792e35d: Status 404 returned error can't find the container with id adf151a9f5eeefe67cd2a14ad01f4eae7e43949325386efd86e050702792e35d Oct 31 22:11:04.211365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.206878 4294 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a1e93389-0027-4e47-adb2-176eef3e5880-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:04.211365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.206917 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtbbf\" (UniqueName: \"kubernetes.io/projected/a1e93389-0027-4e47-adb2-176eef3e5880-kube-api-access-jtbbf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:04.425748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.425714 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg"] Oct 31 22:11:04.426394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.426377 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a1e93389-0027-4e47-adb2-176eef3e5880" containerName="sim" Oct 31 22:11:04.426394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.426391 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e93389-0027-4e47-adb2-176eef3e5880" containerName="sim" Oct 31 22:11:04.426520 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.426510 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e93389-0027-4e47-adb2-176eef3e5880" containerName="sim" Oct 31 22:11:04.429932 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.429914 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.450396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.450372 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg"] Oct 31 22:11:04.514775 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.514710 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.514909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.514786 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.514909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.514888 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69m5z\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-kube-api-access-69m5z\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.515085 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.515067 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.616525 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.616493 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.616704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.616584 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.616704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.616687 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.616833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.616754 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69m5z\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-kube-api-access-69m5z\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.617263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.617245 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.617488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.617468 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.619059 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.619040 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.641058 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.641029 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69m5z\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-kube-api-access-69m5z\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.675544 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.675516 4294 generic.go:356] "Generic (PLEG): container finished" podID="a1e93389-0027-4e47-adb2-176eef3e5880" containerID="fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787" exitCode=0 Oct 31 22:11:04.675710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.675572 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" Oct 31 22:11:04.675710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.675599 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" event={"ID":"a1e93389-0027-4e47-adb2-176eef3e5880","Type":"ContainerDied","Data":"fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787"} Oct 31 22:11:04.675710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.675637 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl" event={"ID":"a1e93389-0027-4e47-adb2-176eef3e5880","Type":"ContainerDied","Data":"3b7f232e3f4627c3377583411c12eeeff7ebd5eb8807743eea10fdab95361034"} Oct 31 22:11:04.675710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.675653 4294 scope.go:122] "RemoveContainer" containerID="fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787" Oct 31 22:11:04.677371 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.677346 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" event={"ID":"5afad580-b8d1-4293-8b21-176a395c3f4d","Type":"ContainerStarted","Data":"8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c"} Oct 31 22:11:04.677500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.677392 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" event={"ID":"5afad580-b8d1-4293-8b21-176a395c3f4d","Type":"ContainerStarted","Data":"adf151a9f5eeefe67cd2a14ad01f4eae7e43949325386efd86e050702792e35d"} Oct 31 22:11:04.677500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.677416 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:04.687274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.687252 4294 scope.go:122] "RemoveContainer" containerID="fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787" Oct 31 22:11:04.687642 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:04.687604 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787\": container with ID starting with fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787 not found: ID does not exist" containerID="fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787" Oct 31 22:11:04.687737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.687656 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787"} err="failed to get container status \"fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787\": rpc error: code = NotFound desc = could not find container \"fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787\": container with ID starting with fdc4be3c7e2049be9edc339d720c80e364ee245c83b64a90311d58c35c754787 not found: ID does not exist" Oct 31 22:11:04.733781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.733737 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" podStartSLOduration=1.7337267029999999 podStartE2EDuration="1.733726703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:04.731733296 +0000 UTC m=+3304.856840527" watchObservedRunningTime="2026-10-31 22:11:04.733726703 +0000 UTC m=+3304.858833933" Oct 31 22:11:04.744266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.744228 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:04.774765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.774692 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl"] Oct 31 22:11:04.779810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:04.779782 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-639a39/e2e-usage-sim-bd9aba-86b8fbc958-lmwbl"] Oct 31 22:11:05.057678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:05.057650 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg"] Oct 31 22:11:05.057791 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:05.057696 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67068587_c7a1_43d0_8604_78d480304345.slice/crio-10ff75402965e7c8b87fe4894adfed57d50f9e897f1572993de0c042f9ef70ca WatchSource:0}: Error finding container 10ff75402965e7c8b87fe4894adfed57d50f9e897f1572993de0c042f9ef70ca: Status 404 returned error can't find the container with id 10ff75402965e7c8b87fe4894adfed57d50f9e897f1572993de0c042f9ef70ca Oct 31 22:11:05.686228 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:05.686193 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" event={"ID":"67068587-c7a1-43d0-8604-78d480304345","Type":"ContainerStarted","Data":"d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3"} Oct 31 22:11:05.686723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:05.686254 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" event={"ID":"67068587-c7a1-43d0-8604-78d480304345","Type":"ContainerStarted","Data":"10ff75402965e7c8b87fe4894adfed57d50f9e897f1572993de0c042f9ef70ca"} Oct 31 22:11:05.686723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:05.686506 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:05.732152 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:05.732100 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" podStartSLOduration=1.7320835319999999 podStartE2EDuration="1.732083532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:05.726722764 +0000 UTC m=+3305.851829995" watchObservedRunningTime="2026-10-31 22:11:05.732083532 +0000 UTC m=+3305.857190764" Oct 31 22:11:05.734704 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:05.734681 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-5ee76dc49b61: secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:05.735296 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:05.735266 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9: secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:05.735434 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:05.735412 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials podName:5afad580-b8d1-4293-8b21-176a395c3f4d nodeName:}" failed. No retries permitted until 2026-10-31 22:11:06.235337804 +0000 UTC m=+3306.360445043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d") : secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:05.758376 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:05.758353 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:11:05.772235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:05.772212 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:11:06.074326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.071528 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e93389-0027-4e47-adb2-176eef3e5880" path="/var/lib/kubelet/pods/a1e93389-0027-4e47-adb2-176eef3e5880/volumes" Oct 31 22:11:06.207326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.177331 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg"] Oct 31 22:11:06.233204 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.233167 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw"] Oct 31 22:11:06.254071 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.247521 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.262469 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:06.262317 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-5ee76dc49b61: secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:06.262469 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:06.262363 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9: secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:06.262469 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:06.262423 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials podName:5afad580-b8d1-4293-8b21-176a395c3f4d nodeName:}" failed. No retries permitted until 2026-10-31 22:11:07.262403882 +0000 UTC m=+3307.387511103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d") : secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:06.263269 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.263242 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9mx\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-kube-api-access-2s9mx\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.263636 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.263613 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.263950 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.263931 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.264198 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.264177 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.270792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.270760 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw"] Oct 31 22:11:06.366278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.366193 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.366472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.366293 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.366472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.366457 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9mx\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-kube-api-access-2s9mx\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.366604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.366545 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.367012 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.366988 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.367225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.367196 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.369416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.369389 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.390010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.389981 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9mx\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-kube-api-access-2s9mx\") pod \"payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.592777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.592741 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:06.710752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.710719 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:11:06.774227 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:06.774191 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-4379f1a3a622: secret "agc-cred-4379f1a3a622" not found Oct 31 22:11:06.774227 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:06.774221 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg: secret "agc-cred-4379f1a3a622" not found Oct 31 22:11:06.774513 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:06.774283 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials podName:67068587-c7a1-43d0-8604-78d480304345 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:07.274265355 +0000 UTC m=+3307.399372583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" (UID: "67068587-c7a1-43d0-8604-78d480304345") : secret "agc-cred-4379f1a3a622" not found Oct 31 22:11:06.897566 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:06.897495 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3889d3ba_7c0a_4f8f_8a4b_79e50c50f2e5.slice/crio-7f8c77a48be43d2e9adb52e864354a9991cc0c1d31925afecc4b55017a7a65ac WatchSource:0}: Error finding container 7f8c77a48be43d2e9adb52e864354a9991cc0c1d31925afecc4b55017a7a65ac: Status 404 returned error can't find the container with id 7f8c77a48be43d2e9adb52e864354a9991cc0c1d31925afecc4b55017a7a65ac Oct 31 22:11:06.897879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:06.897857 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw"] Oct 31 22:11:07.010224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.010197 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9"] Oct 31 22:11:07.041777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.041751 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69"] Oct 31 22:11:07.045982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.045966 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.065187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.065159 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69"] Oct 31 22:11:07.077452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.077419 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.077708 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.077682 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6xd\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-kube-api-access-5l6xd\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.077774 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.077755 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.078066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.078043 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.182233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.182121 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.182233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.182215 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.182531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.182398 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6xd\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-kube-api-access-5l6xd\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.182531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.182456 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.183478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.183451 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.183653 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.183581 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.185238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.185211 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.206741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.206714 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6xd\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-kube-api-access-5l6xd\") pod \"payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.284352 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.284318 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-2c83ed9af069: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:07.284352 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.284350 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-4379f1a3a622: secret "agc-cred-4379f1a3a622" not found Oct 31 22:11:07.284352 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.284361 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg: [secret "agc-cred-2c83ed9af069" not found, secret "agc-cred-4379f1a3a622" not found] Oct 31 22:11:07.284672 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.284413 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials podName:67068587-c7a1-43d0-8604-78d480304345 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:08.284398353 +0000 UTC m=+3308.409505562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" (UID: "67068587-c7a1-43d0-8604-78d480304345") : [secret "agc-cred-2c83ed9af069" not found, secret "agc-cred-4379f1a3a622" not found] Oct 31 22:11:07.284672 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.284480 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-5ee76dc49b61: secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:07.284672 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.284510 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9: secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:07.284672 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.284562 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials podName:5afad580-b8d1-4293-8b21-176a395c3f4d nodeName:}" failed. No retries permitted until 2026-10-31 22:11:09.284547288 +0000 UTC m=+3309.409654508 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d") : secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:07.362021 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.361990 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:07.748386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.746760 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw"] Oct 31 22:11:07.772235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.770373 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8"] Oct 31 22:11:07.785137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.785109 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69"] Oct 31 22:11:07.785838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.785803 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" event={"ID":"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5","Type":"ContainerStarted","Data":"f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3"} Oct 31 22:11:07.785957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.785851 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" event={"ID":"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5","Type":"ContainerStarted","Data":"7f8c77a48be43d2e9adb52e864354a9991cc0c1d31925afecc4b55017a7a65ac"} Oct 31 22:11:07.786137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.785974 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:07.786137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.785992 4294 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" secret="" err="secret \"payload-processing-external-model-e2e--34b55c0e-dockercfg-r5jhw\" not found" Oct 31 22:11:07.836100 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.836076 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-2c83ed9af069: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:07.836100 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.836097 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:07.836294 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.836125 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-plugins-e2e-extmod-b-56a6d4: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:07.836294 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.836139 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:08.3361248 +0000 UTC m=+3308.461232032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:07.836294 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:07.836215 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:08.33619702 +0000 UTC m=+3308.461304245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:07.964324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:07.944622 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" podStartSLOduration=1.944603082 podStartE2EDuration="1.944603082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:07.913822675 +0000 UTC m=+3308.038929907" watchObservedRunningTime="2026-10-31 22:11:07.944603082 +0000 UTC m=+3308.069710313" Oct 31 22:11:08.352379 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352350 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-plugins-e2e-extmod-b-56a6d4: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:08.352548 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352416 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:09.352401896 +0000 UTC m=+3309.477509105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:08.352548 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352464 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-2c83ed9af069: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:08.352548 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352484 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:08.352548 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352522 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:09.352511802 +0000 UTC m=+3309.477619015 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:08.352803 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352563 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-2c83ed9af069: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:08.352803 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352577 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-4379f1a3a622: secret "agc-cred-4379f1a3a622" not found Oct 31 22:11:08.352803 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352585 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg: [secret "agc-cred-2c83ed9af069" not found, secret "agc-cred-4379f1a3a622" not found] Oct 31 22:11:08.352803 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.352622 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials podName:67068587-c7a1-43d0-8604-78d480304345 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:10.35261486 +0000 UTC m=+3310.477722069 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" (UID: "67068587-c7a1-43d0-8604-78d480304345") : [secret "agc-cred-2c83ed9af069" not found, secret "agc-cred-4379f1a3a622" not found] Oct 31 22:11:08.489608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:08.489577 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6"] Oct 31 22:11:08.503527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:08.503498 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69"] Oct 31 22:11:08.794498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:08.794463 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" event={"ID":"871503c6-2b43-4952-bf40-9cb45a298491","Type":"ContainerStarted","Data":"a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b"} Oct 31 22:11:08.794901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:08.794524 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" event={"ID":"871503c6-2b43-4952-bf40-9cb45a298491","Type":"ContainerStarted","Data":"14e595dc7cd933de93c7303b8386408827fc323a0628f6327f01321ff53cd4c7"} Oct 31 22:11:08.794901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:08.794552 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:08.794901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:08.794649 4294 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" secret="" err="secret \"payload-processing-external-model-e2e--1e97e1d3-dockercfg-nkvhj\" not found" Oct 31 22:11:08.938328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:08.934359 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" podStartSLOduration=1.934321373 podStartE2EDuration="1.934321373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:08.873609375 +0000 UTC m=+3308.998716619" watchObservedRunningTime="2026-10-31 22:11:08.934321373 +0000 UTC m=+3309.059428607" Oct 31 22:11:08.964809 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.964778 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-883ce719e9af: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:08.965048 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.965006 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:08.965223 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.965210 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:09.465187623 +0000 UTC m=+3309.590294835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : secret "agc-cred-883ce719e9af" not found Oct 31 22:11:08.965411 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.965391 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-plugins-e2e-extmod-a-943963: configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:08.965502 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:08.965449 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:09.465433346 +0000 UTC m=+3309.590540568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:09.371099 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371068 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-5ee76dc49b61: secret "agc-cred-5ee76dc49b61" not found Oct 31 22:11:09.371099 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371100 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-883ce719e9af: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:09.371333 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371112 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9: [secret "agc-cred-5ee76dc49b61" not found, secret "agc-cred-883ce719e9af" not found] Oct 31 22:11:09.371333 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371153 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-2c83ed9af069: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:09.371333 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371171 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:09.371333 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371175 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials podName:5afad580-b8d1-4293-8b21-176a395c3f4d nodeName:}" failed. No retries permitted until 2026-10-31 22:11:13.371157718 +0000 UTC m=+3313.496264946 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d") : [secret "agc-cred-5ee76dc49b61" not found, secret "agc-cred-883ce719e9af" not found] Oct 31 22:11:09.371333 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371197 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-plugins-e2e-extmod-b-56a6d4: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:09.371333 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371211 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:11.371198649 +0000 UTC m=+3311.496305862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:09.371333 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.371233 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:11.371223512 +0000 UTC m=+3311.496330734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:09.472973 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.472948 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-883ce719e9af: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:09.472973 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.472968 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:09.473124 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.472987 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-plugins-e2e-extmod-a-943963: configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:09.473124 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.473013 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:10.472998732 +0000 UTC m=+3310.598105967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : secret "agc-cred-883ce719e9af" not found Oct 31 22:11:09.473124 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:09.473032 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:10.47302568 +0000 UTC m=+3310.598132889 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:10.386011 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.385981 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-2c83ed9af069: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:10.386011 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.386007 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-4379f1a3a622: secret "agc-cred-4379f1a3a622" not found Oct 31 22:11:10.386011 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.386017 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg: [secret "agc-cred-2c83ed9af069" not found, secret "agc-cred-4379f1a3a622" not found] Oct 31 22:11:10.386481 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.386067 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials podName:67068587-c7a1-43d0-8604-78d480304345 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:14.386054039 +0000 UTC m=+3314.511161278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" (UID: "67068587-c7a1-43d0-8604-78d480304345") : [secret "agc-cred-2c83ed9af069" not found, secret "agc-cred-4379f1a3a622" not found] Oct 31 22:11:10.486985 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.486952 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-883ce719e9af: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:10.486985 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.486980 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:10.487194 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.487000 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-plugins-e2e-extmod-a-943963: configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:10.487194 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.487041 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:12.487025221 +0000 UTC m=+3312.612132448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : secret "agc-cred-883ce719e9af" not found Oct 31 22:11:10.487194 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:10.487058 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:12.487049877 +0000 UTC m=+3312.612157086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:11.404460 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:11.404418 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-56a6d4/agc-cred-2c83ed9af069: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:11.404460 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:11.404453 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw: secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:11.404907 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:11.404483 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-plugins-e2e-extmod-b-56a6d4: configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:11.404907 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:11.404522 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:15.404501954 +0000 UTC m=+3315.529609179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : secret "agc-cred-2c83ed9af069" not found Oct 31 22:11:11.404907 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:11.404538 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume podName:3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:15.40453179 +0000 UTC m=+3315.529639002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5") : configmap "payload-processing-external-model-plugins-e2e-extmod-b-56a6d4" not found Oct 31 22:11:12.011065 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.011011 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" podUID="5afad580-b8d1-4293-8b21-176a395c3f4d" containerName="payload-processing" containerID="cri-o://8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c" gracePeriod=25 Oct 31 22:11:12.398896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.398870 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:12.441322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.439882 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-plugins-config-volume\") pod \"5afad580-b8d1-4293-8b21-176a395c3f4d\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " Oct 31 22:11:12.441322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.440099 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials\") pod \"5afad580-b8d1-4293-8b21-176a395c3f4d\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " Oct 31 22:11:12.441322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.440135 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-routing-overlay-volume\") pod \"5afad580-b8d1-4293-8b21-176a395c3f4d\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " Oct 31 22:11:12.441322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.440356 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-kube-api-access-9wkwd\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-kube-api-access-9wkwd\") pod \"5afad580-b8d1-4293-8b21-176a395c3f4d\" (UID: \"5afad580-b8d1-4293-8b21-176a395c3f4d\") " Oct 31 22:11:12.459751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.459713 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-plugins-config-volume" pod "5afad580-b8d1-4293-8b21-176a395c3f4d" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:12.470581 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.470544 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials" pod "5afad580-b8d1-4293-8b21-176a395c3f4d" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:12.471091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.471069 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-routing-overlay-volume" pod "5afad580-b8d1-4293-8b21-176a395c3f4d" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:12.471386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.471365 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-kube-api-access-9wkwd" pod "5afad580-b8d1-4293-8b21-176a395c3f4d" (UID: "5afad580-b8d1-4293-8b21-176a395c3f4d"). InnerVolumeSpecName "kube-api-access-9wkwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:12.565370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.545820 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.566173 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:12.550494 4294 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-943963/agc-cred-883ce719e9af: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:12.566173 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:12.565692 4294 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69: secret "agc-cred-883ce719e9af" not found Oct 31 22:11:12.566173 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:12.550548 4294 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-plugins-e2e-extmod-a-943963: configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:12.566173 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:12.565778 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:16.565752361 +0000 UTC m=+3316.690859593 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : secret "agc-cred-883ce719e9af" not found Oct 31 22:11:12.566173 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:12.565829 4294 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume podName:871503c6-2b43-4952-bf40-9cb45a298491 nodeName:}" failed. No retries permitted until 2026-10-31 22:11:16.565804258 +0000 UTC m=+3316.690911467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" (UID: "871503c6-2b43-4952-bf40-9cb45a298491") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-943963" not found Oct 31 22:11:12.566633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.566610 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wkwd\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-kube-api-access-9wkwd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.566733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.566721 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5afad580-b8d1-4293-8b21-176a395c3f4d-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.566820 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.566809 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5afad580-b8d1-4293-8b21-176a395c3f4d-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:12.771373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.771326 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" podUID="eafda436-26c7-4192-898b-bba67a024214" containerName="payload-processing" containerID="cri-o://c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0" gracePeriod=25 Oct 31 22:11:12.786638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.786603 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" podUID="67068587-c7a1-43d0-8604-78d480304345" containerName="payload-processing" containerID="cri-o://d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3" gracePeriod=25 Oct 31 22:11:12.831043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.831013 4294 generic.go:356] "Generic (PLEG): container finished" podID="5afad580-b8d1-4293-8b21-176a395c3f4d" containerID="8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c" exitCode=0 Oct 31 22:11:12.831190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.831057 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" Oct 31 22:11:12.831190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.831103 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" event={"ID":"5afad580-b8d1-4293-8b21-176a395c3f4d","Type":"ContainerDied","Data":"8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c"} Oct 31 22:11:12.831190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.831167 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9" event={"ID":"5afad580-b8d1-4293-8b21-176a395c3f4d","Type":"ContainerDied","Data":"adf151a9f5eeefe67cd2a14ad01f4eae7e43949325386efd86e050702792e35d"} Oct 31 22:11:12.831343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.831206 4294 scope.go:122] "RemoveContainer" containerID="8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c" Oct 31 22:11:12.869899 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.869874 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2"] Oct 31 22:11:12.873701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.873678 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-0877f5-wqvc2"] Oct 31 22:11:12.941886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.941859 4294 scope.go:122] "RemoveContainer" containerID="8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c" Oct 31 22:11:12.942211 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:12.942192 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c\": container with ID starting with 8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c not found: ID does not exist" containerID="8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c" Oct 31 22:11:12.942284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.942225 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c"} err="failed to get container status \"8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c\": rpc error: code = NotFound desc = could not find container \"8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c\": container with ID starting with 8788a9665044a55e19d74f2ba537e3ee61c65dd7a86de90f27663fe70efbaf7c not found: ID does not exist" Oct 31 22:11:12.957385 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.957340 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9"] Oct 31 22:11:12.964049 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:12.964012 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-868bb8qq9"] Oct 31 22:11:12.971552 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:12.971529 4294 logging.go:55] [core] [Channel #5726 SubChannel #5727]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.20:50052", ServerName: "10.134.1.20:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.20:50052: connect: connection refused" Oct 31 22:11:13.089993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.089964 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:11:13.183458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.183433 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-kube-api-access-4fwln\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-kube-api-access-4fwln\") pod \"eafda436-26c7-4192-898b-bba67a024214\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " Oct 31 22:11:13.183648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.183627 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-plugins-config-volume\") pod \"eafda436-26c7-4192-898b-bba67a024214\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " Oct 31 22:11:13.183816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.183802 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-provider-credentials\") pod \"eafda436-26c7-4192-898b-bba67a024214\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " Oct 31 22:11:13.184102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.184075 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-plugins-config-volume" pod "eafda436-26c7-4192-898b-bba67a024214" (UID: "eafda436-26c7-4192-898b-bba67a024214"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:13.184397 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.184372 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-routing-overlay-volume\") pod \"eafda436-26c7-4192-898b-bba67a024214\" (UID: \"eafda436-26c7-4192-898b-bba67a024214\") " Oct 31 22:11:13.185155 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.185131 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.185532 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.185508 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-routing-overlay-volume" pod "eafda436-26c7-4192-898b-bba67a024214" (UID: "eafda436-26c7-4192-898b-bba67a024214"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:13.186118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.186093 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-kube-api-access-4fwln" pod "eafda436-26c7-4192-898b-bba67a024214" (UID: "eafda436-26c7-4192-898b-bba67a024214"). InnerVolumeSpecName "kube-api-access-4fwln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:13.186416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.186396 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-provider-credentials" pod "eafda436-26c7-4192-898b-bba67a024214" (UID: "eafda436-26c7-4192-898b-bba67a024214"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:13.201888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.201868 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:13.286691 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.286654 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-kube-api-access-69m5z\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-kube-api-access-69m5z\") pod \"67068587-c7a1-43d0-8604-78d480304345\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " Oct 31 22:11:13.286880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.286784 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials\") pod \"67068587-c7a1-43d0-8604-78d480304345\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " Oct 31 22:11:13.286880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.286827 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-routing-overlay-volume\") pod \"67068587-c7a1-43d0-8604-78d480304345\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " Oct 31 22:11:13.287123 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.287105 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-plugins-config-volume\") pod \"67068587-c7a1-43d0-8604-78d480304345\" (UID: \"67068587-c7a1-43d0-8604-78d480304345\") " Oct 31 22:11:13.287796 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.287773 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.287896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.287798 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eafda436-26c7-4192-898b-bba67a024214-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.287896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.287812 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fwln\" (UniqueName: \"kubernetes.io/projected/eafda436-26c7-4192-898b-bba67a024214-kube-api-access-4fwln\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.288190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.288168 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-plugins-config-volume" pod "67068587-c7a1-43d0-8604-78d480304345" (UID: "67068587-c7a1-43d0-8604-78d480304345"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:13.288529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.288508 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-routing-overlay-volume" pod "67068587-c7a1-43d0-8604-78d480304345" (UID: "67068587-c7a1-43d0-8604-78d480304345"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:13.289606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.289580 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-kube-api-access-69m5z" pod "67068587-c7a1-43d0-8604-78d480304345" (UID: "67068587-c7a1-43d0-8604-78d480304345"). InnerVolumeSpecName "kube-api-access-69m5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:13.289910 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.289892 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials" pod "67068587-c7a1-43d0-8604-78d480304345" (UID: "67068587-c7a1-43d0-8604-78d480304345"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:13.389819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.389735 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69m5z\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-kube-api-access-69m5z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.389819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.389764 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/67068587-c7a1-43d0-8604-78d480304345-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.389819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.389773 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.389819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.389782 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67068587-c7a1-43d0-8604-78d480304345-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:13.490076 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.490033 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" podUID="bf6c0b2b-4ceb-404d-900f-6b351268964c" containerName="payload-processing" containerID="cri-o://17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313" gracePeriod=25 Oct 31 22:11:13.757900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.757874 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:11:13.795320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.795269 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" podUID="3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" containerName="payload-processing" containerID="cri-o://f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3" gracePeriod=25 Oct 31 22:11:13.843787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.843755 4294 generic.go:356] "Generic (PLEG): container finished" podID="67068587-c7a1-43d0-8604-78d480304345" containerID="d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3" exitCode=0 Oct 31 22:11:13.843980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.843819 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" Oct 31 22:11:13.843980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.843842 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" event={"ID":"67068587-c7a1-43d0-8604-78d480304345","Type":"ContainerDied","Data":"d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3"} Oct 31 22:11:13.843980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.843909 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg" event={"ID":"67068587-c7a1-43d0-8604-78d480304345","Type":"ContainerDied","Data":"10ff75402965e7c8b87fe4894adfed57d50f9e897f1572993de0c042f9ef70ca"} Oct 31 22:11:13.843980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.843959 4294 scope.go:122] "RemoveContainer" containerID="d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3" Oct 31 22:11:13.851390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.851365 4294 generic.go:356] "Generic (PLEG): container finished" podID="eafda436-26c7-4192-898b-bba67a024214" containerID="c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0" exitCode=0 Oct 31 22:11:13.851573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.851559 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" Oct 31 22:11:13.852035 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.852012 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" event={"ID":"eafda436-26c7-4192-898b-bba67a024214","Type":"ContainerDied","Data":"c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0"} Oct 31 22:11:13.852230 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.852209 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" event={"ID":"eafda436-26c7-4192-898b-bba67a024214","Type":"ContainerDied","Data":"a3b716e66dcbc279518f05839afb711df78491dce677c4ad227c90150fcfd6e4"} Oct 31 22:11:13.860514 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.860488 4294 generic.go:356] "Generic (PLEG): container finished" podID="bf6c0b2b-4ceb-404d-900f-6b351268964c" containerID="17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313" exitCode=0 Oct 31 22:11:13.860621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.860526 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" Oct 31 22:11:13.860621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.860541 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" event={"ID":"bf6c0b2b-4ceb-404d-900f-6b351268964c","Type":"ContainerDied","Data":"17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313"} Oct 31 22:11:13.860621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.860580 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6" event={"ID":"bf6c0b2b-4ceb-404d-900f-6b351268964c","Type":"ContainerDied","Data":"7596d8367b6dfbc398ee4acf704cef031b8170146efcf90eeefe9ca06ef8d92f"} Oct 31 22:11:13.903777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.903382 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-provider-credentials\") pod \"bf6c0b2b-4ceb-404d-900f-6b351268964c\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " Oct 31 22:11:13.903777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.903472 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-kube-api-access-9jhbl\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-kube-api-access-9jhbl\") pod \"bf6c0b2b-4ceb-404d-900f-6b351268964c\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " Oct 31 22:11:13.903777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.903662 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-plugins-config-volume\") pod \"bf6c0b2b-4ceb-404d-900f-6b351268964c\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " Oct 31 22:11:13.903777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.903710 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-routing-overlay-volume\") pod \"bf6c0b2b-4ceb-404d-900f-6b351268964c\" (UID: \"bf6c0b2b-4ceb-404d-900f-6b351268964c\") " Oct 31 22:11:13.904982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.904954 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-routing-overlay-volume" pod "bf6c0b2b-4ceb-404d-900f-6b351268964c" (UID: "bf6c0b2b-4ceb-404d-900f-6b351268964c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:13.905411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.905385 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-plugins-config-volume" pod "bf6c0b2b-4ceb-404d-900f-6b351268964c" (UID: "bf6c0b2b-4ceb-404d-900f-6b351268964c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:13.906883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.906853 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-kube-api-access-9jhbl" pod "bf6c0b2b-4ceb-404d-900f-6b351268964c" (UID: "bf6c0b2b-4ceb-404d-900f-6b351268964c"). InnerVolumeSpecName "kube-api-access-9jhbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:13.907294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.907272 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-provider-credentials" pod "bf6c0b2b-4ceb-404d-900f-6b351268964c" (UID: "bf6c0b2b-4ceb-404d-900f-6b351268964c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:13.911505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.911481 4294 scope.go:122] "RemoveContainer" containerID="d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3" Oct 31 22:11:13.911912 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:13.911892 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3\": container with ID starting with d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3 not found: ID does not exist" containerID="d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3" Oct 31 22:11:13.912014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.911920 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3"} err="failed to get container status \"d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3\": rpc error: code = NotFound desc = could not find container \"d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3\": container with ID starting with d59e6a54174cf8abf279483019dbb35ac5d16b96a49839444e3f962a8ce68ae3 not found: ID does not exist" Oct 31 22:11:13.912014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.911938 4294 scope.go:122] "RemoveContainer" containerID="c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0" Oct 31 22:11:13.922857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.922839 4294 scope.go:122] "RemoveContainer" containerID="c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0" Oct 31 22:11:13.923176 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:13.923137 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0\": container with ID starting with c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0 not found: ID does not exist" containerID="c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0" Oct 31 22:11:13.923275 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.923184 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0"} err="failed to get container status \"c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0\": rpc error: code = NotFound desc = could not find container \"c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0\": container with ID starting with c4b04f9ae28ff6d276e7a154956f2aa0ee0f4242f11de10b4d5287df4337afc0 not found: ID does not exist" Oct 31 22:11:13.923275 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.923206 4294 scope.go:122] "RemoveContainer" containerID="17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313" Oct 31 22:11:13.931261 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.931237 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg"] Oct 31 22:11:13.940427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.938078 4294 scope.go:122] "RemoveContainer" containerID="17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313" Oct 31 22:11:13.940427 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:13.938453 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313\": container with ID starting with 17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313 not found: ID does not exist" containerID="17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313" Oct 31 22:11:13.940427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.938487 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313"} err="failed to get container status \"17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313\": rpc error: code = NotFound desc = could not find container \"17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313\": container with ID starting with 17359b53edc3a01267a435f2261a0c698505ea094e33348097a098d43c899313 not found: ID does not exist" Oct 31 22:11:13.940427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.940215 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6cb74zdpg"] Oct 31 22:11:13.975662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.975617 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8" podUID="eafda436-26c7-4192-898b-bba67a024214" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.1.20:50052\" within 1s: context deadline exceeded" Oct 31 22:11:13.975868 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:13.975777 4294 logging.go:55] [core] [Channel #5726 SubChannel #5727]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.1.20:50052", ServerName: "10.134.1.20:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.1.20:50052: operation was canceled" Oct 31 22:11:13.976717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.976689 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8"] Oct 31 22:11:13.994260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.994227 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5afad580-b8d1-4293-8b21-176a395c3f4d" path="/var/lib/kubelet/pods/5afad580-b8d1-4293-8b21-176a395c3f4d/volumes" Oct 31 22:11:13.995196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.995175 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67068587-c7a1-43d0-8604-78d480304345" path="/var/lib/kubelet/pods/67068587-c7a1-43d0-8604-78d480304345/volumes" Oct 31 22:11:13.995991 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.995962 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994e4e92-f1c1-4f8c-8440-bfe426a248ed" path="/var/lib/kubelet/pods/994e4e92-f1c1-4f8c-8440-bfe426a248ed/volumes" Oct 31 22:11:13.996654 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:13.996624 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-78d47jdp8"] Oct 31 22:11:14.007871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.006361 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.007871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.006396 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jhbl\" (UniqueName: \"kubernetes.io/projected/bf6c0b2b-4ceb-404d-900f-6b351268964c-kube-api-access-9jhbl\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.007871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.006413 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.007871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.006427 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bf6c0b2b-4ceb-404d-900f-6b351268964c-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.187117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.187043 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:14.254556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.254514 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6"] Oct 31 22:11:14.277324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.274530 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-585fqsgb6"] Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.314340 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume\") pod \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.314411 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials\") pod \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.314483 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-kube-api-access-2s9mx\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-kube-api-access-2s9mx\") pod \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.314613 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-routing-overlay-volume\") pod \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\" (UID: \"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5\") " Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.314862 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume" pod "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.315462 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.315806 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-routing-overlay-volume" pod "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:14.318651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.318259 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-kube-api-access-2s9mx" pod "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5"). InnerVolumeSpecName "kube-api-access-2s9mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:14.351325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.343556 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials" pod "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" (UID: "3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:14.417174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.417140 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.417174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.417169 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.417174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.417180 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s9mx\" (UniqueName: \"kubernetes.io/projected/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5-kube-api-access-2s9mx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:14.802681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.802640 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" podUID="871503c6-2b43-4952-bf40-9cb45a298491" containerName="payload-processing" containerID="cri-o://a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b" gracePeriod=25 Oct 31 22:11:14.992091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.992055 4294 generic.go:356] "Generic (PLEG): container finished" podID="3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" containerID="f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3" exitCode=0 Oct 31 22:11:14.992280 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.992197 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" event={"ID":"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5","Type":"ContainerDied","Data":"f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3"} Oct 31 22:11:14.992280 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.992244 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" event={"ID":"3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5","Type":"ContainerDied","Data":"7f8c77a48be43d2e9adb52e864354a9991cc0c1d31925afecc4b55017a7a65ac"} Oct 31 22:11:14.992280 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.992273 4294 scope.go:122] "RemoveContainer" containerID="f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3" Oct 31 22:11:14.992600 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:14.992399 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw" Oct 31 22:11:15.030134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.030100 4294 scope.go:122] "RemoveContainer" containerID="f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3" Oct 31 22:11:15.030583 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:15.030538 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3\": container with ID starting with f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3 not found: ID does not exist" containerID="f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3" Oct 31 22:11:15.030665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.030577 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3"} err="failed to get container status \"f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3\": rpc error: code = NotFound desc = could not find container \"f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3\": container with ID starting with f570b07fcd6f5f99acb19454f45a6eea5f786f6741b1b6edf1f461ebec2702e3 not found: ID does not exist" Oct 31 22:11:15.084500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.084265 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw"] Oct 31 22:11:15.093990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.093941 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-56a6d4/payload-processing-external-model-e2e-extmod-b-56a6d4-6f4chgkrw"] Oct 31 22:11:15.194576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.194543 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:15.338079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.337951 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume\") pod \"871503c6-2b43-4952-bf40-9cb45a298491\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " Oct 31 22:11:15.338079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.338029 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials\") pod \"871503c6-2b43-4952-bf40-9cb45a298491\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " Oct 31 22:11:15.338290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.338176 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-kube-api-access-5l6xd\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-kube-api-access-5l6xd\") pod \"871503c6-2b43-4952-bf40-9cb45a298491\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " Oct 31 22:11:15.338290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.338242 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-routing-overlay-volume\") pod \"871503c6-2b43-4952-bf40-9cb45a298491\" (UID: \"871503c6-2b43-4952-bf40-9cb45a298491\") " Oct 31 22:11:15.338635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.338610 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume" pod "871503c6-2b43-4952-bf40-9cb45a298491" (UID: "871503c6-2b43-4952-bf40-9cb45a298491"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:15.338989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.338965 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:15.339420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.339397 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-routing-overlay-volume" pod "871503c6-2b43-4952-bf40-9cb45a298491" (UID: "871503c6-2b43-4952-bf40-9cb45a298491"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:15.341595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.341574 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials" pod "871503c6-2b43-4952-bf40-9cb45a298491" (UID: "871503c6-2b43-4952-bf40-9cb45a298491"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:15.341699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.341652 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-kube-api-access-5l6xd" pod "871503c6-2b43-4952-bf40-9cb45a298491" (UID: "871503c6-2b43-4952-bf40-9cb45a298491"). InnerVolumeSpecName "kube-api-access-5l6xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:15.440869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.440836 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:15.440869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.440863 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6xd\" (UniqueName: \"kubernetes.io/projected/871503c6-2b43-4952-bf40-9cb45a298491-kube-api-access-5l6xd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:15.440869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.440876 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/871503c6-2b43-4952-bf40-9cb45a298491-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:15.987010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.986976 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" path="/var/lib/kubelet/pods/3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5/volumes" Oct 31 22:11:15.987467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.987451 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6c0b2b-4ceb-404d-900f-6b351268964c" path="/var/lib/kubelet/pods/bf6c0b2b-4ceb-404d-900f-6b351268964c/volumes" Oct 31 22:11:15.987839 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:15.987823 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafda436-26c7-4192-898b-bba67a024214" path="/var/lib/kubelet/pods/eafda436-26c7-4192-898b-bba67a024214/volumes" Oct 31 22:11:16.016384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.016347 4294 generic.go:356] "Generic (PLEG): container finished" podID="871503c6-2b43-4952-bf40-9cb45a298491" containerID="a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b" exitCode=0 Oct 31 22:11:16.016529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.016380 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" event={"ID":"871503c6-2b43-4952-bf40-9cb45a298491","Type":"ContainerDied","Data":"a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b"} Oct 31 22:11:16.016529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.016411 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" Oct 31 22:11:16.016529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.016469 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69" event={"ID":"871503c6-2b43-4952-bf40-9cb45a298491","Type":"ContainerDied","Data":"14e595dc7cd933de93c7303b8386408827fc323a0628f6327f01321ff53cd4c7"} Oct 31 22:11:16.016529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.016499 4294 scope.go:122] "RemoveContainer" containerID="a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b" Oct 31 22:11:16.026722 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.026704 4294 scope.go:122] "RemoveContainer" containerID="a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b" Oct 31 22:11:16.027031 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:16.027001 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b\": container with ID starting with a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b not found: ID does not exist" containerID="a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b" Oct 31 22:11:16.027128 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.027029 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b"} err="failed to get container status \"a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b\": rpc error: code = NotFound desc = could not find container \"a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b\": container with ID starting with a1b896f9a3987e04329d2bd4cbd69c83a8ad830edd42f3078e8dd71321add16b not found: ID does not exist" Oct 31 22:11:16.070655 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.070623 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69"] Oct 31 22:11:16.081134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.081099 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-943963/payload-processing-external-model-e2e-extmod-a-943963-6cb6j6g69"] Oct 31 22:11:16.890113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.890073 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8"] Oct 31 22:11:16.891052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891030 4294 cpu_manager.go:414] "containerMap: removing container" podUID="67068587-c7a1-43d0-8604-78d480304345" containerName="payload-processing" Oct 31 22:11:16.891109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891057 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67068587-c7a1-43d0-8604-78d480304345" containerName="payload-processing" Oct 31 22:11:16.891109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891073 4294 cpu_manager.go:414] "containerMap: removing container" podUID="871503c6-2b43-4952-bf40-9cb45a298491" containerName="payload-processing" Oct 31 22:11:16.891109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891081 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871503c6-2b43-4952-bf40-9cb45a298491" containerName="payload-processing" Oct 31 22:11:16.891109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891092 4294 cpu_manager.go:414] "containerMap: removing container" podUID="eafda436-26c7-4192-898b-bba67a024214" containerName="payload-processing" Oct 31 22:11:16.891109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891100 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafda436-26c7-4192-898b-bba67a024214" containerName="payload-processing" Oct 31 22:11:16.891357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891121 4294 cpu_manager.go:414] "containerMap: removing container" podUID="bf6c0b2b-4ceb-404d-900f-6b351268964c" containerName="payload-processing" Oct 31 22:11:16.891357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891128 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6c0b2b-4ceb-404d-900f-6b351268964c" containerName="payload-processing" Oct 31 22:11:16.891357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891141 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5afad580-b8d1-4293-8b21-176a395c3f4d" containerName="payload-processing" Oct 31 22:11:16.891357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891155 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5afad580-b8d1-4293-8b21-176a395c3f4d" containerName="payload-processing" Oct 31 22:11:16.891357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891164 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" containerName="payload-processing" Oct 31 22:11:16.891357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891171 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" containerName="payload-processing" Oct 31 22:11:16.891679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891400 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="67068587-c7a1-43d0-8604-78d480304345" containerName="payload-processing" Oct 31 22:11:16.891679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891417 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="871503c6-2b43-4952-bf40-9cb45a298491" containerName="payload-processing" Oct 31 22:11:16.891679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891425 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafda436-26c7-4192-898b-bba67a024214" containerName="payload-processing" Oct 31 22:11:16.891679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891438 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6c0b2b-4ceb-404d-900f-6b351268964c" containerName="payload-processing" Oct 31 22:11:16.891679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891447 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5afad580-b8d1-4293-8b21-176a395c3f4d" containerName="payload-processing" Oct 31 22:11:16.891679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.891456 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3889d3ba-7c0a-4f8f-8a4b-79e50c50f2e5" containerName="payload-processing" Oct 31 22:11:16.896849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.896821 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.916239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.916209 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8"] Oct 31 22:11:16.922194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.922156 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1b0cce-openshift-default-dockercfg-fwx8z\"" Oct 31 22:11:16.964143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964103 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b1fb25-a674-4ce4-917c-964978046bdb-istio-podinfo\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964277 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-credential-socket\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964401 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b1fb25-a674-4ce4-917c-964978046bdb-istiod-ca-cert\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964449 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-envoy\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964563 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-socket\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964629 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-certs\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964674 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkc9h\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-kube-api-access-mkc9h\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964699 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-istio-token\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:16.964931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:16.964787 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-data\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.089729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.089691 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-socket\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.090347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.090319 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-certs\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.090512 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.090493 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkc9h\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-kube-api-access-mkc9h\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.090632 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.090616 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-istio-token\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.090846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.090829 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-data\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.091010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.090996 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b1fb25-a674-4ce4-917c-964978046bdb-istio-podinfo\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.091244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.091227 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-credential-socket\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.091462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.091442 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b1fb25-a674-4ce4-917c-964978046bdb-istiod-ca-cert\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.113328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.111413 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-envoy\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.113535 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.113381 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-certs\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.113671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.113639 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-socket\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.114610 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.114578 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-credential-socket\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.114879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.114851 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-data\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.118324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.115049 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b1fb25-a674-4ce4-917c-964978046bdb-istiod-ca-cert\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.132773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.132740 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-envoy\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.136324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.135055 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b1fb25-a674-4ce4-917c-964978046bdb-istio-podinfo\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.151361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.146781 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-istio-token\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.151361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.148766 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkc9h\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-kube-api-access-mkc9h\") pod \"e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.217216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.216880 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:17.555054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.555021 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8"] Oct 31 22:11:17.556254 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:17.556224 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b1fb25_a674_4ce4_917c_964978046bdb.slice/crio-18265c1dc5644904baf4b3be720ed701bc8e045dc23b4d9d6d9b9afef663ed01 WatchSource:0}: Error finding container 18265c1dc5644904baf4b3be720ed701bc8e045dc23b4d9d6d9b9afef663ed01: Status 404 returned error can't find the container with id 18265c1dc5644904baf4b3be720ed701bc8e045dc23b4d9d6d9b9afef663ed01 Oct 31 22:11:17.559822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.559788 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:11:17.559940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.559872 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:11:17.559940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.559918 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:11:17.988803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:17.988769 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871503c6-2b43-4952-bf40-9cb45a298491" path="/var/lib/kubelet/pods/871503c6-2b43-4952-bf40-9cb45a298491/volumes" Oct 31 22:11:18.090951 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:18.090905 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" event={"ID":"01b1fb25-a674-4ce4-917c-964978046bdb","Type":"ContainerStarted","Data":"a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa"} Oct 31 22:11:18.091482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:18.090995 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" event={"ID":"01b1fb25-a674-4ce4-917c-964978046bdb","Type":"ContainerStarted","Data":"18265c1dc5644904baf4b3be720ed701bc8e045dc23b4d9d6d9b9afef663ed01"} Oct 31 22:11:18.161861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:18.161799 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podStartSLOduration=2.16178361 podStartE2EDuration="2.16178361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:18.15792923 +0000 UTC m=+3318.283036461" watchObservedRunningTime="2026-10-31 22:11:18.16178361 +0000 UTC m=+3318.286890840" Oct 31 22:11:18.218040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:18.218003 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:18.224770 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:18.224735 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:19.100203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:19.100168 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:19.101196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:19.101175 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:11:20.388411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:20.388378 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd"] Oct 31 22:11:20.388881 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:20.388628 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" containerID="cri-o://050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7" gracePeriod=30 Oct 31 22:11:20.989922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:20.989883 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv"] Oct 31 22:11:20.994229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:20.994208 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.014346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.014270 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv"] Oct 31 22:11:21.025740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.025701 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:11:21.025993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.025710 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:11:21.095127 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.095080 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.095374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.095250 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.095516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.095497 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4hf7\" (UniqueName: \"kubernetes.io/projected/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-kube-api-access-x4hf7\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.197295 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.197263 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.197476 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.197434 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4hf7\" (UniqueName: \"kubernetes.io/projected/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-kube-api-access-x4hf7\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.197593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.197569 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.198068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.198046 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.199955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.199929 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.221213 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.221190 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4hf7\" (UniqueName: \"kubernetes.io/projected/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-kube-api-access-x4hf7\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.317994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.317915 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:21.625365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:21.625338 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv"] Oct 31 22:11:21.626283 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:21.626255 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c08736_c7e3_4790_8e0c_bf2a70c5856e.slice/crio-7340c730f0743017dffe6f019fe2c0517f348412f1dfcb55ae3958e79ec81a5f WatchSource:0}: Error finding container 7340c730f0743017dffe6f019fe2c0517f348412f1dfcb55ae3958e79ec81a5f: Status 404 returned error can't find the container with id 7340c730f0743017dffe6f019fe2c0517f348412f1dfcb55ae3958e79ec81a5f Oct 31 22:11:22.121731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:22.121691 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" event={"ID":"b6c08736-c7e3-4790-8e0c-bf2a70c5856e","Type":"ContainerStarted","Data":"b9157412e999d1b850ed433be43b78586cdd7398937958b1725d9d47db851807"} Oct 31 22:11:22.121905 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:22.121750 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" event={"ID":"b6c08736-c7e3-4790-8e0c-bf2a70c5856e","Type":"ContainerStarted","Data":"7340c730f0743017dffe6f019fe2c0517f348412f1dfcb55ae3958e79ec81a5f"} Oct 31 22:11:22.163490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:22.163409 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" podStartSLOduration=2.163395562 podStartE2EDuration="2.163395562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:22.161560775 +0000 UTC m=+3322.286668005" watchObservedRunningTime="2026-10-31 22:11:22.163395562 +0000 UTC m=+3322.288502792" Oct 31 22:11:23.130317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:23.130276 4294 generic.go:356] "Generic (PLEG): container finished" podID="b6c08736-c7e3-4790-8e0c-bf2a70c5856e" containerID="b9157412e999d1b850ed433be43b78586cdd7398937958b1725d9d47db851807" exitCode=0 Oct 31 22:11:23.130825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:23.130350 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" event={"ID":"b6c08736-c7e3-4790-8e0c-bf2a70c5856e","Type":"ContainerDied","Data":"b9157412e999d1b850ed433be43b78586cdd7398937958b1725d9d47db851807"} Oct 31 22:11:24.317447 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.317421 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:24.343484 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.343457 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-xks-ca\") pod \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " Oct 31 22:11:24.343619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.343525 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-openshift-ca\") pod \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " Oct 31 22:11:24.343759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.343740 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-kube-api-access-x4hf7\" (UniqueName: \"kubernetes.io/projected/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-kube-api-access-x4hf7\") pod \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\" (UID: \"b6c08736-c7e3-4790-8e0c-bf2a70c5856e\") " Oct 31 22:11:24.343971 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.343945 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-openshift-ca" pod "b6c08736-c7e3-4790-8e0c-bf2a70c5856e" (UID: "b6c08736-c7e3-4790-8e0c-bf2a70c5856e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:24.344259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.344235 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:24.345688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.345668 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-kube-api-access-x4hf7" pod "b6c08736-c7e3-4790-8e0c-bf2a70c5856e" (UID: "b6c08736-c7e3-4790-8e0c-bf2a70c5856e"). InnerVolumeSpecName "kube-api-access-x4hf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:24.345753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.345680 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-xks-ca" pod "b6c08736-c7e3-4790-8e0c-bf2a70c5856e" (UID: "b6c08736-c7e3-4790-8e0c-bf2a70c5856e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:11:24.445134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.445060 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:24.445134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:24.445088 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4hf7\" (UniqueName: \"kubernetes.io/projected/b6c08736-c7e3-4790-8e0c-bf2a70c5856e-kube-api-access-x4hf7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.146730 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.146691 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" Oct 31 22:11:25.146941 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.146735 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv" event={"ID":"b6c08736-c7e3-4790-8e0c-bf2a70c5856e","Type":"ContainerDied","Data":"7340c730f0743017dffe6f019fe2c0517f348412f1dfcb55ae3958e79ec81a5f"} Oct 31 22:11:25.146941 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.146792 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7340c730f0743017dffe6f019fe2c0517f348412f1dfcb55ae3958e79ec81a5f" Oct 31 22:11:25.669573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.669510 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 22:11:25.762984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.762948 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-podinfo\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.762998 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-data\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763079 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-certs\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763114 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-kube-api-access-gw7r9\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-kube-api-access-gw7r9\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763152 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef9228e-2e5c-4114-8187-423aa04fcae1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9228e-2e5c-4114-8187-423aa04fcae1-istiod-ca-cert\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763236 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-envoy\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763267 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-token\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-token\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763289 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-data" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:25.763472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763327 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-credential-socket\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763574 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-credential-socket" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:25.763693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763582 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-socket\") pod \"1ef9228e-2e5c-4114-8187-423aa04fcae1\" (UID: \"1ef9228e-2e5c-4114-8187-423aa04fcae1\") " Oct 31 22:11:25.763693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763674 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef9228e-2e5c-4114-8187-423aa04fcae1-istiod-ca-cert" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:25.763936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.763913 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-certs" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:25.764107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.764082 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-socket" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:25.764368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.764346 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.764478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.764372 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.764478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.764387 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.764478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.764401 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef9228e-2e5c-4114-8187-423aa04fcae1-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.764478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.764414 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.766693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.766668 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-envoy" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:25.766978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.766956 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-podinfo" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:11:25.767068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.766986 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-token" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:25.767370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.767352 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-kube-api-access-gw7r9" pod "1ef9228e-2e5c-4114-8187-423aa04fcae1" (UID: "1ef9228e-2e5c-4114-8187-423aa04fcae1"). InnerVolumeSpecName "kube-api-access-gw7r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:25.865823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.865789 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.865823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.865819 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.865823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.865828 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef9228e-2e5c-4114-8187-423aa04fcae1-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:25.866054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:25.865837 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7r9\" (UniqueName: \"kubernetes.io/projected/1ef9228e-2e5c-4114-8187-423aa04fcae1-kube-api-access-gw7r9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:26.157481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.157447 4294 generic.go:356] "Generic (PLEG): container finished" podID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerID="050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7" exitCode=0 Oct 31 22:11:26.157665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.157504 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" Oct 31 22:11:26.157665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.157516 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" event={"ID":"1ef9228e-2e5c-4114-8187-423aa04fcae1","Type":"ContainerDied","Data":"050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7"} Oct 31 22:11:26.157665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.157574 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd" event={"ID":"1ef9228e-2e5c-4114-8187-423aa04fcae1","Type":"ContainerDied","Data":"ba75e6a28971b5c9d17f27db745581952938575a7eeffa2266149171c833aac2"} Oct 31 22:11:26.157665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.157595 4294 scope.go:122] "RemoveContainer" containerID="050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7" Oct 31 22:11:26.167990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.167952 4294 scope.go:122] "RemoveContainer" containerID="c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e" Oct 31 22:11:26.177357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.177341 4294 scope.go:122] "RemoveContainer" containerID="050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7" Oct 31 22:11:26.177650 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:26.177627 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7\": container with ID starting with 050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7 not found: ID does not exist" containerID="050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7" Oct 31 22:11:26.177755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.177656 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7"} err="failed to get container status \"050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7\": rpc error: code = NotFound desc = could not find container \"050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7\": container with ID starting with 050ca749cc6f7308e3f7c2a412955659b7723416e050b72ee686adf7f43e2da7 not found: ID does not exist" Oct 31 22:11:26.177755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.177674 4294 scope.go:122] "RemoveContainer" containerID="c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e" Oct 31 22:11:26.177977 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:26.177942 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e\": container with ID starting with c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e not found: ID does not exist" containerID="c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e" Oct 31 22:11:26.178023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.177987 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e"} err="failed to get container status \"c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e\": rpc error: code = NotFound desc = could not find container \"c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e\": container with ID starting with c74a0ad91577a3a81fbdfe859c8b1da72e5473a2df79a8a52672750cb834538e not found: ID does not exist" Oct 31 22:11:26.212392 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.212361 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd"] Oct 31 22:11:26.224510 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:26.224489 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-0877f5-openshift-default-65f9674c97-p9xsd"] Oct 31 22:11:27.985879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:27.985851 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" path="/var/lib/kubelet/pods/1ef9228e-2e5c-4114-8187-423aa04fcae1/volumes" Oct 31 22:11:31.312110 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:31.312068 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:11:31.622795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.622704 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz"] Oct 31 22:11:31.623487 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.623464 4294 cpu_manager.go:414] "containerMap: removing container" podUID="b6c08736-c7e3-4790-8e0c-bf2a70c5856e" containerName="revoke-keys" Oct 31 22:11:31.623487 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.623487 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c08736-c7e3-4790-8e0c-bf2a70c5856e" containerName="revoke-keys" Oct 31 22:11:31.623657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.623523 4294 cpu_manager.go:414] "containerMap: removing container" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" Oct 31 22:11:31.623657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.623531 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" Oct 31 22:11:31.623757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.623694 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" Oct 31 22:11:31.623757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.623710 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" Oct 31 22:11:31.623757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.623732 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c08736-c7e3-4790-8e0c-bf2a70c5856e" containerName="revoke-keys" Oct 31 22:11:31.629019 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.628999 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.647500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.647471 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-710927-openshift-default-dockercfg-rm4nm\"" Oct 31 22:11:31.649729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.649694 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz"] Oct 31 22:11:31.758969 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.758933 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-envoy\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759119 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-credential-socket\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759224 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-data\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759285 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-podinfo\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759361 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fd7\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-kube-api-access-w9fd7\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759389 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-token\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759517 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-socket\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759622 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istiod-ca-cert\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.759989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.759763 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-certs\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.860952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.860914 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-data\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.860961 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-podinfo\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861004 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fd7\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-kube-api-access-w9fd7\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861032 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-token\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861062 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-socket\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861145 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istiod-ca-cert\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861236 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-certs\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861264 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-envoy\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861398 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-credential-socket\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861474 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-data\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861708 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-credential-socket\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861771 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-socket\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.861972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.861948 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-certs\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.862026 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.862002 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istiod-ca-cert\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.863559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.863536 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-envoy\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.863654 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.863637 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-podinfo\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.884822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.884755 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-token\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.884933 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.884877 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fd7\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-kube-api-access-w9fd7\") pod \"e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:31.943643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:31.943603 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:32.312083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:32.312048 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz"] Oct 31 22:11:32.320954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:32.320112 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:11:32.320954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:32.320180 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:11:32.320954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:32.320208 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:11:33.286391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:33.286348 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" event={"ID":"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e","Type":"ContainerStarted","Data":"9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a"} Oct 31 22:11:33.286674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:33.286650 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" event={"ID":"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e","Type":"ContainerStarted","Data":"489cff26bd4b318f726f0b94e4ca0d430c13740efa6d50d85f7d2f761aacc49c"} Oct 31 22:11:33.335555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:33.335504 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podStartSLOduration=2.335489192 podStartE2EDuration="2.335489192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:11:33.333566736 +0000 UTC m=+3333.458673967" watchObservedRunningTime="2026-10-31 22:11:33.335489192 +0000 UTC m=+3333.460596422" Oct 31 22:11:33.944788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:33.944753 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:33.946236 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:33.946210 4294 patch_prober.go:29] interesting pod/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 31 22:11:33.946385 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:33.946267 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 31 22:11:34.944518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:34.944485 4294 patch_prober.go:29] interesting pod/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 31 22:11:34.944923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:34.944533 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 31 22:11:35.944358 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:35.944294 4294 patch_prober.go:29] interesting pod/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 31 22:11:35.944569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:35.944378 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 31 22:11:36.944804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:36.944765 4294 patch_prober.go:29] interesting pod/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 31 22:11:36.945209 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:36.944820 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 31 22:11:37.944842 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:37.944807 4294 patch_prober.go:29] interesting pod/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 31 22:11:37.945223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:37.944867 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 31 22:11:38.949424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:38.949392 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:38.949907 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:38.949874 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:38.950789 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:38.950766 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:11:39.349757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:39.349725 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv"] Oct 31 22:11:39.353231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:39.353206 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-5237cd-w6cfv"] Oct 31 22:11:39.994136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:39.994095 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c08736-c7e3-4790-8e0c-bf2a70c5856e" path="/var/lib/kubelet/pods/b6c08736-c7e3-4790-8e0c-bf2a70c5856e/volumes" Oct 31 22:11:46.037097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:46.037063 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq"] Oct 31 22:11:46.049010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:46.048986 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67d715ad-kjcdq"] Oct 31 22:11:46.950662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:46.950628 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw"] Oct 31 22:11:46.950886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:46.950850 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" containerID="cri-o://8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b" gracePeriod=30 Oct 31 22:11:47.710000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.709960 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h"] Oct 31 22:11:47.710620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.710605 4294 cpu_manager.go:414] "containerMap: removing container" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" Oct 31 22:11:47.710620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.710621 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef9228e-2e5c-4114-8187-423aa04fcae1" containerName="istio-proxy" Oct 31 22:11:47.715586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.715565 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.731650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.731619 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h"] Oct 31 22:11:47.738452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.738424 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:11:47.738452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.738446 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:11:47.784999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.784945 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvvzg\" (UniqueName: \"kubernetes.io/projected/450309f8-0645-4cb8-80ed-834a13530e15-kube-api-access-nvvzg\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.785186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.785090 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.785186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.785155 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.887047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.887011 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvvzg\" (UniqueName: \"kubernetes.io/projected/450309f8-0645-4cb8-80ed-834a13530e15-kube-api-access-nvvzg\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.887232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.887104 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.887232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.887166 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.887878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.887859 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.889503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.889481 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.910601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.910578 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvvzg\" (UniqueName: \"kubernetes.io/projected/450309f8-0645-4cb8-80ed-834a13530e15-kube-api-access-nvvzg\") pod \"maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:47.985501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:47.985468 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76056de0-2080-4ff1-87d3-4aa034d6bd76" path="/var/lib/kubelet/pods/76056de0-2080-4ff1-87d3-4aa034d6bd76/volumes" Oct 31 22:11:48.038136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:48.038111 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:48.342945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:48.342913 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h"] Oct 31 22:11:48.343361 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:11:48.343338 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450309f8_0645_4cb8_80ed_834a13530e15.slice/crio-fcd6b348e8ad064f2fd5a478d72048cb1f7c23a006fdb8975b00f510516f0241 WatchSource:0}: Error finding container fcd6b348e8ad064f2fd5a478d72048cb1f7c23a006fdb8975b00f510516f0241: Status 404 returned error can't find the container with id fcd6b348e8ad064f2fd5a478d72048cb1f7c23a006fdb8975b00f510516f0241 Oct 31 22:11:48.427352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:48.427325 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" event={"ID":"450309f8-0645-4cb8-80ed-834a13530e15","Type":"ContainerStarted","Data":"fcd6b348e8ad064f2fd5a478d72048cb1f7c23a006fdb8975b00f510516f0241"} Oct 31 22:11:49.436439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:49.436397 4294 patch_prober.go:29] interesting pod/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:11:49.436895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:49.436456 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:11:49.436895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:49.436554 4294 generic.go:356] "Generic (PLEG): container finished" podID="450309f8-0645-4cb8-80ed-834a13530e15" containerID="b6360e12f197b0b857dfa72d69fcad312b3e6ef9baa2063bcfba5078cee9ceca" exitCode=0 Oct 31 22:11:49.436895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:49.436643 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" event={"ID":"450309f8-0645-4cb8-80ed-834a13530e15","Type":"ContainerDied","Data":"b6360e12f197b0b857dfa72d69fcad312b3e6ef9baa2063bcfba5078cee9ceca"} Oct 31 22:11:50.605513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.605488 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:50.726444 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.726365 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-xks-ca\") pod \"450309f8-0645-4cb8-80ed-834a13530e15\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " Oct 31 22:11:50.726604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.726453 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450309f8-0645-4cb8-80ed-834a13530e15-kube-api-access-nvvzg\" (UniqueName: \"kubernetes.io/projected/450309f8-0645-4cb8-80ed-834a13530e15-kube-api-access-nvvzg\") pod \"450309f8-0645-4cb8-80ed-834a13530e15\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " Oct 31 22:11:50.726604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.726476 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-openshift-ca\") pod \"450309f8-0645-4cb8-80ed-834a13530e15\" (UID: \"450309f8-0645-4cb8-80ed-834a13530e15\") " Oct 31 22:11:50.726889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.726862 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-openshift-ca" pod "450309f8-0645-4cb8-80ed-834a13530e15" (UID: "450309f8-0645-4cb8-80ed-834a13530e15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:50.727194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.727172 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:50.728599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.728574 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-xks-ca" pod "450309f8-0645-4cb8-80ed-834a13530e15" (UID: "450309f8-0645-4cb8-80ed-834a13530e15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:11:50.728699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.728680 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450309f8-0645-4cb8-80ed-834a13530e15-kube-api-access-nvvzg" pod "450309f8-0645-4cb8-80ed-834a13530e15" (UID: "450309f8-0645-4cb8-80ed-834a13530e15"). InnerVolumeSpecName "kube-api-access-nvvzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:50.828979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.828943 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450309f8-0645-4cb8-80ed-834a13530e15-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:50.828979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:50.828972 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvvzg\" (UniqueName: \"kubernetes.io/projected/450309f8-0645-4cb8-80ed-834a13530e15-kube-api-access-nvvzg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:51.453319 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:51.453278 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" Oct 31 22:11:51.453641 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:51.453328 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h" event={"ID":"450309f8-0645-4cb8-80ed-834a13530e15","Type":"ContainerDied","Data":"fcd6b348e8ad064f2fd5a478d72048cb1f7c23a006fdb8975b00f510516f0241"} Oct 31 22:11:51.453757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:51.453675 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fcd6b348e8ad064f2fd5a478d72048cb1f7c23a006fdb8975b00f510516f0241" Oct 31 22:11:52.365418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.365383 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 22:11:52.454487 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.454449 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs"] Oct 31 22:11:52.465753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.465267 4294 generic.go:356] "Generic (PLEG): container finished" podID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerID="8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b" exitCode=0 Oct 31 22:11:52.465753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.465346 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" event={"ID":"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e","Type":"ContainerDied","Data":"8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b"} Oct 31 22:11:52.465753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.465390 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" event={"ID":"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e","Type":"ContainerDied","Data":"4c840822488fa8ad355fcefbee64d20d8e6eb6df5a55f8c344e5dce514f59a2f"} Oct 31 22:11:52.465753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.465413 4294 scope.go:122] "RemoveContainer" containerID="8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b" Oct 31 22:11:52.465753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.465551 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw" Oct 31 22:11:52.472505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.472475 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-data\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.472651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.472560 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-podinfo\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.472728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.472656 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-token\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.473066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.472766 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istiod-ca-cert\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.473066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.472796 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-socket\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.473066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.472910 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-kube-api-access-swcsd\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-kube-api-access-swcsd\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.473066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.473004 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-envoy\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.473349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.473185 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-certs\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.473349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.473250 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-credential-socket\") pod \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\" (UID: \"5fbfdbbd-4a73-4b68-86c9-ac418fceb78e\") " Oct 31 22:11:52.473349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.473329 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istiod-ca-cert" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:11:52.474847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.473623 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-data" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:52.474847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.473974 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.474847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.474009 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.474847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.474166 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-socket" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:52.474847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.474431 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-certs" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:52.474847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.474605 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-credential-socket" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:52.483657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.483639 4294 scope.go:122] "RemoveContainer" containerID="771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e" Oct 31 22:11:52.484199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.484174 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-token" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:52.486937 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.485521 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-envoy" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:11:52.486937 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.486112 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-kube-api-access-swcsd" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "kube-api-access-swcsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:11:52.486937 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.486741 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-podinfo" pod "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" (UID: "5fbfdbbd-4a73-4b68-86c9-ac418fceb78e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:11:52.520978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.520955 4294 scope.go:122] "RemoveContainer" containerID="8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b" Oct 31 22:11:52.521348 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:52.521325 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b\": container with ID starting with 8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b not found: ID does not exist" containerID="8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b" Oct 31 22:11:52.521466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.521356 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b"} err="failed to get container status \"8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b\": rpc error: code = NotFound desc = could not find container \"8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b\": container with ID starting with 8d22b7656365d02f7b97bc85f3152daedcacf599664763ae514158d3810b467b not found: ID does not exist" Oct 31 22:11:52.521466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.521374 4294 scope.go:122] "RemoveContainer" containerID="771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e" Oct 31 22:11:52.521670 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:11:52.521651 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e\": container with ID starting with 771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e not found: ID does not exist" containerID="771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e" Oct 31 22:11:52.521723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.521675 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e"} err="failed to get container status \"771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e\": rpc error: code = NotFound desc = could not find container \"771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e\": container with ID starting with 771c77c829d12fb8dfe55eac9336de56c76f84ba1e7effc0f0ede148d0c13c7e not found: ID does not exist" Oct 31 22:11:52.577442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.577403 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.577442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.577444 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swcsd\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-kube-api-access-swcsd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.577802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.577459 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.577802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.577471 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.577802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.577486 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.577802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.577500 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.577802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.577513 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:11:52.819004 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.818970 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw"] Oct 31 22:11:52.827382 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:52.827346 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-5237cd-openshift-default-5f58b4f7bf-xlcmw"] Oct 31 22:11:54.004017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:54.003977 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" path="/var/lib/kubelet/pods/5fbfdbbd-4a73-4b68-86c9-ac418fceb78e/volumes" Oct 31 22:11:58.034022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:58.033984 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs"] Oct 31 22:11:58.045534 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:58.045505 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2705900-fq7rs"] Oct 31 22:11:59.989092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:11:59.989052 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d" path="/var/lib/kubelet/pods/0f8dcd09-22d9-41b5-8b29-7fa9ae6f4f1d/volumes" Oct 31 22:12:01.312282 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:12:01.312219 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:12:03.521613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.521531 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 31 22:12:03.521613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.521598 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 31 22:12:03.607907 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.607872 4294 generic.go:356] "Generic (PLEG): container finished" podID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerID="f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347" exitCode=0 Oct 31 22:12:03.608107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.607947 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" event={"ID":"6a1c20cf-b63c-4934-b0e9-485b2df551f4","Type":"ContainerDied","Data":"f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347"} Oct 31 22:12:03.608882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.608856 4294 scope.go:122] "RemoveContainer" containerID="f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347" Oct 31 22:12:03.611403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.611365 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:03.611499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.611484 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:03.611549 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:03.611535 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:04.616961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:04.616926 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" event={"ID":"6a1c20cf-b63c-4934-b0e9-485b2df551f4","Type":"ContainerStarted","Data":"1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce"} Oct 31 22:12:05.390327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:05.389709 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:12:05.390327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:05.389836 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 31 22:12:05.390327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:05.389896 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 31 22:12:06.382331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:06.381712 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 31 22:12:06.382331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:06.381779 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 31 22:12:07.393327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.392740 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 31 22:12:07.393327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.392816 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 31 22:12:07.497451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.497410 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" containerID="cri-o://62dfca15931dda99286d518bd4308ff462a3cfee4e522de5eeee7ac79bfcfef7" gracePeriod=45 Oct 31 22:12:07.643901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.643804 4294 generic.go:356] "Generic (PLEG): container finished" podID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerID="62dfca15931dda99286d518bd4308ff462a3cfee4e522de5eeee7ac79bfcfef7" exitCode=0 Oct 31 22:12:07.643901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.643879 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" event={"ID":"51a9eca7-b8a4-4428-a645-359a8bddb7b2","Type":"ContainerDied","Data":"62dfca15931dda99286d518bd4308ff462a3cfee4e522de5eeee7ac79bfcfef7"} Oct 31 22:12:07.843896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.843865 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 22:12:07.948642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.948562 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-home\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-home\") pod \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " Oct 31 22:12:07.948828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.948808 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tls-certs\") pod \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " Oct 31 22:12:07.948905 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.948826 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-home" pod "51a9eca7-b8a4-4428-a645-359a8bddb7b2" (UID: "51a9eca7-b8a4-4428-a645-359a8bddb7b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.949027 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.948991 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-model-cache\") pod \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " Oct 31 22:12:07.949101 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949056 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-dshm\") pod \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " Oct 31 22:12:07.949139 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949119 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tmp-dir\") pod \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " Oct 31 22:12:07.949173 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949146 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51a9eca7-b8a4-4428-a645-359a8bddb7b2-kube-api-access-6hbk5\" (UniqueName: \"kubernetes.io/projected/51a9eca7-b8a4-4428-a645-359a8bddb7b2-kube-api-access-6hbk5\") pod \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\" (UID: \"51a9eca7-b8a4-4428-a645-359a8bddb7b2\") " Oct 31 22:12:07.949528 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949379 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-model-cache" pod "51a9eca7-b8a4-4428-a645-359a8bddb7b2" (UID: "51a9eca7-b8a4-4428-a645-359a8bddb7b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.949645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949591 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tmp-dir" pod "51a9eca7-b8a4-4428-a645-359a8bddb7b2" (UID: "51a9eca7-b8a4-4428-a645-359a8bddb7b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:07.949945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949921 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:07.949945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949943 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:07.950086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.949952 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:07.951624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.951600 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tls-certs" pod "51a9eca7-b8a4-4428-a645-359a8bddb7b2" (UID: "51a9eca7-b8a4-4428-a645-359a8bddb7b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:12:07.952292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.952263 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51a9eca7-b8a4-4428-a645-359a8bddb7b2-kube-api-access-6hbk5" pod "51a9eca7-b8a4-4428-a645-359a8bddb7b2" (UID: "51a9eca7-b8a4-4428-a645-359a8bddb7b2"). InnerVolumeSpecName "kube-api-access-6hbk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:12:07.952834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:07.952807 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-dshm" pod "51a9eca7-b8a4-4428-a645-359a8bddb7b2" (UID: "51a9eca7-b8a4-4428-a645-359a8bddb7b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:12:08.051746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.051714 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51a9eca7-b8a4-4428-a645-359a8bddb7b2-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.051746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.051743 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51a9eca7-b8a4-4428-a645-359a8bddb7b2-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.051945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.051756 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hbk5\" (UniqueName: \"kubernetes.io/projected/51a9eca7-b8a4-4428-a645-359a8bddb7b2-kube-api-access-6hbk5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:08.388325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.382666 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 31 22:12:08.388325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.382741 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 31 22:12:08.653723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.653636 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" Oct 31 22:12:08.653723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.653669 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" event={"ID":"51a9eca7-b8a4-4428-a645-359a8bddb7b2","Type":"ContainerDied","Data":"f5bfb591a9fb5690020fb926778a5b8da7a4f00ae0fc2fbbd381e85957dab5b7"} Oct 31 22:12:08.653723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.653727 4294 scope.go:122] "RemoveContainer" containerID="62dfca15931dda99286d518bd4308ff462a3cfee4e522de5eeee7ac79bfcfef7" Oct 31 22:12:08.710771 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.710739 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs"] Oct 31 22:12:08.721697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.721669 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs"] Oct 31 22:12:08.796427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:08.796376 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4b5989/distinct-w0-4b5989-kserve-7cd4644dc4-h6pqs" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 22:12:09.384756 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:09.384717 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" start-of-body= Oct 31 22:12:09.385061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:09.385028 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.44:15021/healthz/ready\": dial tcp 10.134.1.44:15021: connect: connection refused" Oct 31 22:12:09.986224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:09.986195 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" path="/var/lib/kubelet/pods/51a9eca7-b8a4-4428-a645-359a8bddb7b2/volumes" Oct 31 22:12:10.381023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:10.380941 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:12:10.381436 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:10.381414 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:12:10.382148 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:10.382128 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:12:18.468809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.468739 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn"] Oct 31 22:12:18.469498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469479 4294 cpu_manager.go:414] "containerMap: removing container" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" Oct 31 22:12:18.469568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469499 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" Oct 31 22:12:18.469568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469524 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" Oct 31 22:12:18.469568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469530 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" Oct 31 22:12:18.469568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469536 4294 cpu_manager.go:414] "containerMap: removing container" podUID="450309f8-0645-4cb8-80ed-834a13530e15" containerName="revoke-keys" Oct 31 22:12:18.469568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469541 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450309f8-0645-4cb8-80ed-834a13530e15" containerName="revoke-keys" Oct 31 22:12:18.469568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469561 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" Oct 31 22:12:18.469568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469566 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" Oct 31 22:12:18.469792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469670 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="51a9eca7-b8a4-4428-a645-359a8bddb7b2" containerName="main" Oct 31 22:12:18.469792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469681 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" Oct 31 22:12:18.469792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.469687 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="450309f8-0645-4cb8-80ed-834a13530e15" containerName="revoke-keys" Oct 31 22:12:18.473000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.472984 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.488175 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.488152 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c2d1b8-openshift-default-dockercfg-7999k\"" Oct 31 22:12:18.506262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.506232 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn"] Oct 31 22:12:18.589231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589203 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-istio-token\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589246 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-credential-socket\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589367 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-data\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589489 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589474 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beffe07e-cd89-436c-bad9-914094c2c732-istiod-ca-cert\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589525 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-certs\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589615 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589600 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-envoy\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589653 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589629 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp5b4\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-kube-api-access-hp5b4\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589669 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beffe07e-cd89-436c-bad9-914094c2c732-istio-podinfo\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.589766 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.589753 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-socket\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.690633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690602 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-certs\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.690791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690652 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-envoy\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.690791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690677 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp5b4\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-kube-api-access-hp5b4\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.690791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690722 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beffe07e-cd89-436c-bad9-914094c2c732-istio-podinfo\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.690791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690769 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-socket\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690819 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-istio-token\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690857 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-credential-socket\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690891 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-data\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.690976 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beffe07e-cd89-436c-bad9-914094c2c732-istiod-ca-cert\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691237 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.691054 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-certs\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.691263 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-socket\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.691340 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-credential-socket\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691449 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.691377 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-data\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.691802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.691778 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beffe07e-cd89-436c-bad9-914094c2c732-istiod-ca-cert\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.692745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.692726 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-envoy\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.693184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.693167 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beffe07e-cd89-436c-bad9-914094c2c732-istio-podinfo\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.713624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.713601 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp5b4\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-kube-api-access-hp5b4\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.713624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.713672 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-istio-token\") pod \"e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:18.786687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:18.786658 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:19.104419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.104385 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn"] Oct 31 22:12:19.106103 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:12:19.106080 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeffe07e_cd89_436c_bad9_914094c2c732.slice/crio-48bcedd9f71ddfdd0355d7d66c51435026fc3fb6c6032738a37371d8dd3b29c4 WatchSource:0}: Error finding container 48bcedd9f71ddfdd0355d7d66c51435026fc3fb6c6032738a37371d8dd3b29c4: Status 404 returned error can't find the container with id 48bcedd9f71ddfdd0355d7d66c51435026fc3fb6c6032738a37371d8dd3b29c4 Oct 31 22:12:19.108919 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.108874 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:19.109017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.108948 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:19.109017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.108980 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:19.737495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.737462 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" event={"ID":"beffe07e-cd89-436c-bad9-914094c2c732","Type":"ContainerStarted","Data":"3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214"} Oct 31 22:12:19.748226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.748188 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" event={"ID":"beffe07e-cd89-436c-bad9-914094c2c732","Type":"ContainerStarted","Data":"48bcedd9f71ddfdd0355d7d66c51435026fc3fb6c6032738a37371d8dd3b29c4"} Oct 31 22:12:19.786920 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.786886 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:19.798174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:19.798125 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podStartSLOduration=1.798111475 podStartE2EDuration="1.798111475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:12:19.793933303 +0000 UTC m=+3379.919040535" watchObservedRunningTime="2026-10-31 22:12:19.798111475 +0000 UTC m=+3379.923218706" Oct 31 22:12:20.795214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:20.795175 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 31 22:12:20.796452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:20.796403 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 31 22:12:21.790325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:21.789614 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 31 22:12:21.790325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:21.789672 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 31 22:12:21.966209 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:21.966180 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq"] Oct 31 22:12:21.966948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:21.966933 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbfdbbd-4a73-4b68-86c9-ac418fceb78e" containerName="istio-proxy" Oct 31 22:12:21.970634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:21.970617 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.002334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:21.998559 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:12:22.002334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:21.998735 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:12:22.015855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.013599 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq"] Oct 31 22:12:22.015855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.013680 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn"] Oct 31 22:12:22.018499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.018477 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.034386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.034365 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn"] Oct 31 22:12:22.035290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.035197 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv7hm\" (UniqueName: \"kubernetes.io/projected/82988d9b-07ca-4675-a72a-58b290f7cfcb-kube-api-access-qv7hm\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.035449 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.035346 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.035449 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.035435 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.136710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.136634 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.136710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.136687 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.136904 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.136786 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv7hm\" (UniqueName: \"kubernetes.io/projected/82988d9b-07ca-4675-a72a-58b290f7cfcb-kube-api-access-qv7hm\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.136904 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.136844 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.137015 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.136949 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.137072 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.137035 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btp94\" (UniqueName: \"kubernetes.io/projected/223b7c62-329f-4557-b2f1-5d5cef390870-kube-api-access-btp94\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.137561 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.137534 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.139297 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.139275 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.160025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.159998 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv7hm\" (UniqueName: \"kubernetes.io/projected/82988d9b-07ca-4675-a72a-58b290f7cfcb-kube-api-access-qv7hm\") pod \"maas-api-revoke-sub-277359d2-hpcfq\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.238122 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.238094 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.238278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.238146 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.238374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.238346 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btp94\" (UniqueName: \"kubernetes.io/projected/223b7c62-329f-4557-b2f1-5d5cef390870-kube-api-access-btp94\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.238778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.238722 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.240626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.240604 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.261425 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.261404 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btp94\" (UniqueName: \"kubernetes.io/projected/223b7c62-329f-4557-b2f1-5d5cef390870-kube-api-access-btp94\") pod \"maas-api-revoke-sub-c2a4f08d-wglrn\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.319481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.319443 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:22.334841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.334811 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:22.695381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.693393 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq"] Oct 31 22:12:22.707579 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:12:22.707547 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82988d9b_07ca_4675_a72a_58b290f7cfcb.slice/crio-546afb4811f84bba061ea0de93a50a86e4f815b06710b29acf86a090ee03ea98 WatchSource:0}: Error finding container 546afb4811f84bba061ea0de93a50a86e4f815b06710b29acf86a090ee03ea98: Status 404 returned error can't find the container with id 546afb4811f84bba061ea0de93a50a86e4f815b06710b29acf86a090ee03ea98 Oct 31 22:12:22.711311 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.711272 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn"] Oct 31 22:12:22.712508 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:12:22.712483 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223b7c62_329f_4557_b2f1_5d5cef390870.slice/crio-d2219ea9d7cbc85d726d64537700bc5e45535a3e55b683a72e4d2f17a43fe0d0 WatchSource:0}: Error finding container d2219ea9d7cbc85d726d64537700bc5e45535a3e55b683a72e4d2f17a43fe0d0: Status 404 returned error can't find the container with id d2219ea9d7cbc85d726d64537700bc5e45535a3e55b683a72e4d2f17a43fe0d0 Oct 31 22:12:22.758911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.758883 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" event={"ID":"223b7c62-329f-4557-b2f1-5d5cef390870","Type":"ContainerStarted","Data":"d2219ea9d7cbc85d726d64537700bc5e45535a3e55b683a72e4d2f17a43fe0d0"} Oct 31 22:12:22.760160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.760138 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" event={"ID":"82988d9b-07ca-4675-a72a-58b290f7cfcb","Type":"ContainerStarted","Data":"546afb4811f84bba061ea0de93a50a86e4f815b06710b29acf86a090ee03ea98"} Oct 31 22:12:22.793326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.790411 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 31 22:12:22.793326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:22.790471 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 31 22:12:23.780871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:23.774268 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" event={"ID":"82988d9b-07ca-4675-a72a-58b290f7cfcb","Type":"ContainerStarted","Data":"20e82cbf45467b85a5c08db65677ea9fe0df42e4c188ad16bfa00b940afe4cf8"} Oct 31 22:12:23.814351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:23.813482 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 31 22:12:23.814351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:23.813539 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 31 22:12:23.817335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:23.816434 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" event={"ID":"223b7c62-329f-4557-b2f1-5d5cef390870","Type":"ContainerStarted","Data":"88432bd681c6b2a290a8456d2e16290f514a1e6b4a9590a83d267aee9bd369c6"} Oct 31 22:12:23.825340 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:23.824997 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" podStartSLOduration=2.824979662 podStartE2EDuration="2.824979662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:12:23.823679578 +0000 UTC m=+3383.948786814" watchObservedRunningTime="2026-10-31 22:12:23.824979662 +0000 UTC m=+3383.950086893" Oct 31 22:12:24.795410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:24.794889 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 31 22:12:24.795410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:24.794951 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 31 22:12:24.825649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:24.825613 4294 generic.go:356] "Generic (PLEG): container finished" podID="223b7c62-329f-4557-b2f1-5d5cef390870" containerID="88432bd681c6b2a290a8456d2e16290f514a1e6b4a9590a83d267aee9bd369c6" exitCode=0 Oct 31 22:12:24.825867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:24.825703 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" event={"ID":"223b7c62-329f-4557-b2f1-5d5cef390870","Type":"ContainerDied","Data":"88432bd681c6b2a290a8456d2e16290f514a1e6b4a9590a83d267aee9bd369c6"} Oct 31 22:12:24.827713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:24.827688 4294 generic.go:356] "Generic (PLEG): container finished" podID="82988d9b-07ca-4675-a72a-58b290f7cfcb" containerID="20e82cbf45467b85a5c08db65677ea9fe0df42e4c188ad16bfa00b940afe4cf8" exitCode=0 Oct 31 22:12:24.827851 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:24.827838 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" event={"ID":"82988d9b-07ca-4675-a72a-58b290f7cfcb","Type":"ContainerDied","Data":"20e82cbf45467b85a5c08db65677ea9fe0df42e4c188ad16bfa00b940afe4cf8"} Oct 31 22:12:25.787908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:25.787870 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body= Oct 31 22:12:25.788100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:25.787927 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" Oct 31 22:12:26.058809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.058778 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:26.084054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.084025 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:26.120724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.120692 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-openshift-ca\") pod \"223b7c62-329f-4557-b2f1-5d5cef390870\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " Oct 31 22:12:26.120909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.120804 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-xks-ca\") pod \"223b7c62-329f-4557-b2f1-5d5cef390870\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " Oct 31 22:12:26.120909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.120826 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82988d9b-07ca-4675-a72a-58b290f7cfcb-kube-api-access-qv7hm\" (UniqueName: \"kubernetes.io/projected/82988d9b-07ca-4675-a72a-58b290f7cfcb-kube-api-access-qv7hm\") pod \"82988d9b-07ca-4675-a72a-58b290f7cfcb\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " Oct 31 22:12:26.120909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.120863 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-openshift-ca\") pod \"82988d9b-07ca-4675-a72a-58b290f7cfcb\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " Oct 31 22:12:26.121089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.120924 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-xks-ca\") pod \"82988d9b-07ca-4675-a72a-58b290f7cfcb\" (UID: \"82988d9b-07ca-4675-a72a-58b290f7cfcb\") " Oct 31 22:12:26.121089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.120982 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/223b7c62-329f-4557-b2f1-5d5cef390870-kube-api-access-btp94\" (UniqueName: \"kubernetes.io/projected/223b7c62-329f-4557-b2f1-5d5cef390870-kube-api-access-btp94\") pod \"223b7c62-329f-4557-b2f1-5d5cef390870\" (UID: \"223b7c62-329f-4557-b2f1-5d5cef390870\") " Oct 31 22:12:26.121195 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.121135 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-openshift-ca" pod "223b7c62-329f-4557-b2f1-5d5cef390870" (UID: "223b7c62-329f-4557-b2f1-5d5cef390870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:12:26.121542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.121515 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-openshift-ca" pod "82988d9b-07ca-4675-a72a-58b290f7cfcb" (UID: "82988d9b-07ca-4675-a72a-58b290f7cfcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:12:26.121662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.121611 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:26.121662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.121626 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:26.123409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.123381 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-xks-ca" pod "82988d9b-07ca-4675-a72a-58b290f7cfcb" (UID: "82988d9b-07ca-4675-a72a-58b290f7cfcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:12:26.123643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.123622 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-xks-ca" pod "223b7c62-329f-4557-b2f1-5d5cef390870" (UID: "223b7c62-329f-4557-b2f1-5d5cef390870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:12:26.123944 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.123920 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82988d9b-07ca-4675-a72a-58b290f7cfcb-kube-api-access-qv7hm" pod "82988d9b-07ca-4675-a72a-58b290f7cfcb" (UID: "82988d9b-07ca-4675-a72a-58b290f7cfcb"). InnerVolumeSpecName "kube-api-access-qv7hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:12:26.124039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.124021 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223b7c62-329f-4557-b2f1-5d5cef390870-kube-api-access-btp94" pod "223b7c62-329f-4557-b2f1-5d5cef390870" (UID: "223b7c62-329f-4557-b2f1-5d5cef390870"). InnerVolumeSpecName "kube-api-access-btp94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:12:26.222883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.222845 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82988d9b-07ca-4675-a72a-58b290f7cfcb-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:26.222883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.222880 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btp94\" (UniqueName: \"kubernetes.io/projected/223b7c62-329f-4557-b2f1-5d5cef390870-kube-api-access-btp94\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:26.223103 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.222892 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/223b7c62-329f-4557-b2f1-5d5cef390870-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:26.223103 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.222906 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv7hm\" (UniqueName: \"kubernetes.io/projected/82988d9b-07ca-4675-a72a-58b290f7cfcb-kube-api-access-qv7hm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:12:26.792686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.792652 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:26.793088 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.793066 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:26.793772 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.793754 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:12:26.853251 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.853210 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" event={"ID":"223b7c62-329f-4557-b2f1-5d5cef390870","Type":"ContainerDied","Data":"d2219ea9d7cbc85d726d64537700bc5e45535a3e55b683a72e4d2f17a43fe0d0"} Oct 31 22:12:26.853483 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.853232 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn" Oct 31 22:12:26.853483 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.853353 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2219ea9d7cbc85d726d64537700bc5e45535a3e55b683a72e4d2f17a43fe0d0" Oct 31 22:12:26.854988 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.854964 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" Oct 31 22:12:26.855156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.854987 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq" event={"ID":"82988d9b-07ca-4675-a72a-58b290f7cfcb","Type":"ContainerDied","Data":"546afb4811f84bba061ea0de93a50a86e4f815b06710b29acf86a090ee03ea98"} Oct 31 22:12:26.855156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:26.855044 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="546afb4811f84bba061ea0de93a50a86e4f815b06710b29acf86a090ee03ea98" Oct 31 22:12:31.328213 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:12:31.328176 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:12:35.030916 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:35.030876 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5"] Oct 31 22:12:35.042611 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:35.042580 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ee3803f-k2wc5"] Oct 31 22:12:35.989781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:35.989743 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411b519c-ad33-4941-ba57-4e8fe32c8614" path="/var/lib/kubelet/pods/411b519c-ad33-4941-ba57-4e8fe32c8614/volumes" Oct 31 22:12:41.990518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:41.990487 4294 generic.go:356] "Generic (PLEG): container finished" podID="01b1fb25-a674-4ce4-917c-964978046bdb" containerID="a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa" exitCode=0 Oct 31 22:12:41.990874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:41.990549 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" event={"ID":"01b1fb25-a674-4ce4-917c-964978046bdb","Type":"ContainerDied","Data":"a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa"} Oct 31 22:12:41.990916 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:41.990871 4294 scope.go:122] "RemoveContainer" containerID="a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa" Oct 31 22:12:41.992599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:41.992569 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:41.992662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:41.992641 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:41.992701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:41.992675 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:12:43.005546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:43.005509 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" event={"ID":"01b1fb25-a674-4ce4-917c-964978046bdb","Type":"ContainerStarted","Data":"3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6"} Oct 31 22:12:43.218507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:43.218468 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:12:43.218688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:43.218644 4294 patch_prober.go:29] interesting pod/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 31 22:12:43.218738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:43.218716 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 31 22:12:44.224496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:44.224037 4294 patch_prober.go:29] interesting pod/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 31 22:12:44.224496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:44.224099 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 31 22:12:45.194925 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.193152 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2"] Oct 31 22:12:45.194925 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.194123 4294 cpu_manager.go:414] "containerMap: removing container" podUID="82988d9b-07ca-4675-a72a-58b290f7cfcb" containerName="revoke-keys" Oct 31 22:12:45.194925 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.194146 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82988d9b-07ca-4675-a72a-58b290f7cfcb" containerName="revoke-keys" Oct 31 22:12:45.202345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.199527 4294 cpu_manager.go:414] "containerMap: removing container" podUID="223b7c62-329f-4557-b2f1-5d5cef390870" containerName="revoke-keys" Oct 31 22:12:45.202345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.199549 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="223b7c62-329f-4557-b2f1-5d5cef390870" containerName="revoke-keys" Oct 31 22:12:45.202345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.199770 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="82988d9b-07ca-4675-a72a-58b290f7cfcb" containerName="revoke-keys" Oct 31 22:12:45.202345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.199798 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="223b7c62-329f-4557-b2f1-5d5cef390870" containerName="revoke-keys" Oct 31 22:12:45.203070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.203044 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.217018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.216634 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2"] Oct 31 22:12:45.219803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.219777 4294 patch_prober.go:29] interesting pod/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 31 22:12:45.219934 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.219833 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 31 22:12:45.224672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.224647 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c08360\"/\"default-dockercfg-8lpck\"" Oct 31 22:12:45.225050 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.224711 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c08360\"/\"test-model-c08360-kserve-self-signed-certs\"" Oct 31 22:12:45.230130 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.230107 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c08360\"/\"openshift-service-ca.crt\"" Oct 31 22:12:45.230249 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.230150 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c08360\"/\"kube-root-ca.crt\"" Oct 31 22:12:45.292351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.292242 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-dshm\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.292542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.292395 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-model-cache\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.292618 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.292527 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdf7\" (UniqueName: \"kubernetes.io/projected/a77ca828-2884-4809-b2a7-9ead63055834-kube-api-access-ppdf7\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.292618 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.292606 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-tmp-dir\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.292745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.292649 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-home\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.292745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.292680 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77ca828-2884-4809-b2a7-9ead63055834-tls-certs\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.394322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.394258 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdf7\" (UniqueName: \"kubernetes.io/projected/a77ca828-2884-4809-b2a7-9ead63055834-kube-api-access-ppdf7\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.394647 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.394620 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-tmp-dir\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.394810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.394793 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-home\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.394933 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.394918 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77ca828-2884-4809-b2a7-9ead63055834-tls-certs\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.395185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.395165 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-dshm\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.395378 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.395361 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-model-cache\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.396028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.396007 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-model-cache\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.401320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.399453 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-home\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.401320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.399734 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-tmp-dir\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.402849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.402826 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77ca828-2884-4809-b2a7-9ead63055834-tls-certs\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.407860 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.407826 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-dshm\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.429485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.429453 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdf7\" (UniqueName: \"kubernetes.io/projected/a77ca828-2884-4809-b2a7-9ead63055834-kube-api-access-ppdf7\") pod \"test-model-c08360-kserve-7d8985c679-lnqz2\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:45.523327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:45.521568 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:46.102817 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:46.102781 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2"] Oct 31 22:12:46.103665 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:12:46.103624 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda77ca828_2884_4809_b2a7_9ead63055834.slice/crio-5fb7ae5fc6772e3f15cbde022c15324e5ec62f62941f9af5fe37e655da0d06ec WatchSource:0}: Error finding container 5fb7ae5fc6772e3f15cbde022c15324e5ec62f62941f9af5fe37e655da0d06ec: Status 404 returned error can't find the container with id 5fb7ae5fc6772e3f15cbde022c15324e5ec62f62941f9af5fe37e655da0d06ec Oct 31 22:12:46.218638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:46.218610 4294 patch_prober.go:29] interesting pod/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 31 22:12:46.218764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:46.218661 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 31 22:12:47.036707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:47.036659 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" event={"ID":"a77ca828-2884-4809-b2a7-9ead63055834","Type":"ContainerStarted","Data":"ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b"} Oct 31 22:12:47.036707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:47.036714 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" event={"ID":"a77ca828-2884-4809-b2a7-9ead63055834","Type":"ContainerStarted","Data":"5fb7ae5fc6772e3f15cbde022c15324e5ec62f62941f9af5fe37e655da0d06ec"} Oct 31 22:12:47.094324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:47.090700 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" podStartSLOduration=2.090678613 podStartE2EDuration="2.090678613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:12:47.08776872 +0000 UTC m=+3407.212875980" watchObservedRunningTime="2026-10-31 22:12:47.090678613 +0000 UTC m=+3407.215785863" Oct 31 22:12:47.223342 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:47.219699 4294 patch_prober.go:29] interesting pod/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 31 22:12:47.223342 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:47.219773 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 31 22:12:47.525433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:47.525393 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:48.217875 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:48.217839 4294 patch_prober.go:29] interesting pod/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" start-of-body= Oct 31 22:12:48.218583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:48.218548 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.51:15021/healthz/ready\": dial tcp 10.134.1.51:15021: connect: connection refused" Oct 31 22:12:49.100661 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:49.100635 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:12:49.223431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:49.223401 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:12:50.077699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:50.077665 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:12:55.522329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:55.522270 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:55.535342 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:55.535315 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:12:56.122895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:12:56.122861 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:13:01.312756 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:13:01.312704 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:13:03.296978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:03.296942 4294 scope.go:122] "RemoveContainer" containerID="8a3114c4bd3efbc04be02fa7430f9f5f3139d886cb2626d3ece85d79161b4a58" Oct 31 22:13:03.308013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:03.307994 4294 scope.go:122] "RemoveContainer" containerID="f9230f060a77e432f3f421215d5855262f3add1c10db87ffd1a90624e8fc8b62" Oct 31 22:13:08.672063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:08.672027 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9"] Oct 31 22:13:08.672569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:08.672264 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" podUID="2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" containerName="maas-api" containerID="cri-o://bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95" gracePeriod=30 Oct 31 22:13:09.152335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.147275 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 22:13:09.208325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.206362 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-openshift-service-ca\") pod \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " Oct 31 22:13:09.208325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.206504 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-metrics-certs\") pod \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " Oct 31 22:13:09.208325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.206564 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-kube-api-access-cvnmv\" (UniqueName: \"kubernetes.io/projected/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-kube-api-access-cvnmv\") pod \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " Oct 31 22:13:09.208325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.206666 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-maas-api-tls\") pod \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\" (UID: \"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea\") " Oct 31 22:13:09.212576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.209034 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-openshift-service-ca" pod "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" (UID: "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:09.236015 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.235941 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-kube-api-access-cvnmv" pod "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" (UID: "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea"). InnerVolumeSpecName "kube-api-access-cvnmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:09.236506 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.236292 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-maas-api-tls" pod "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" (UID: "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:13:09.240938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.239170 4294 generic.go:356] "Generic (PLEG): container finished" podID="2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" containerID="bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95" exitCode=0 Oct 31 22:13:09.240938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.239456 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" event={"ID":"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea","Type":"ContainerDied","Data":"bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95"} Oct 31 22:13:09.240938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.239519 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" event={"ID":"2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea","Type":"ContainerDied","Data":"507219e89515d9be59cd397d9258a9ce809ff38bce9136aa36aba7c8cddd24fe"} Oct 31 22:13:09.240938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.239551 4294 scope.go:122] "RemoveContainer" containerID="bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95" Oct 31 22:13:09.240938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.239705 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-metrics-certs" pod "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" (UID: "2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:13:09.240938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.239726 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9" Oct 31 22:13:09.270040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.270015 4294 scope.go:122] "RemoveContainer" containerID="bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95" Oct 31 22:13:09.270469 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:13:09.270440 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95\": container with ID starting with bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95 not found: ID does not exist" containerID="bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95" Oct 31 22:13:09.270569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.270470 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95"} err="failed to get container status \"bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95\": rpc error: code = NotFound desc = could not find container \"bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95\": container with ID starting with bb86795203b00b22e62f5a3f657ea02fec85d4b18dded20f9d228102fc709c95 not found: ID does not exist" Oct 31 22:13:09.296467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.296414 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9"] Oct 31 22:13:09.299924 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.299902 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-4b5989-5999fb45ff-zdzr9"] Oct 31 22:13:09.310098 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.310070 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnmv\" (UniqueName: \"kubernetes.io/projected/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-kube-api-access-cvnmv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:09.310098 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.310100 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:09.310292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.310116 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:09.310292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.310128 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:09.442752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.442724 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h"] Oct 31 22:13:09.446611 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.446589 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4b5989-7nz6h"] Oct 31 22:13:09.988017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.987976 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" path="/var/lib/kubelet/pods/2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea/volumes" Oct 31 22:13:09.988626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:09.988606 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450309f8-0645-4cb8-80ed-834a13530e15" path="/var/lib/kubelet/pods/450309f8-0645-4cb8-80ed-834a13530e15/volumes" Oct 31 22:13:16.572457 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:16.572422 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f"] Oct 31 22:13:16.572955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:16.572685 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" containerID="cri-o://90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1" gracePeriod=30 Oct 31 22:13:21.034612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.034574 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867"] Oct 31 22:13:21.046259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.046228 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acb0e2de-nv867"] Oct 31 22:13:21.454233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.454140 4294 patch_prober.go:29] interesting pod/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:13:21.454233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.454222 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:13:21.867943 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.867914 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 22:13:21.989595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.989556 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb59d8a-a11d-4371-a69c-de170a7dd517" path="/var/lib/kubelet/pods/ceb59d8a-a11d-4371-a69c-de170a7dd517/volumes" Oct 31 22:13:21.991913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.991891 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-envoy\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.991950 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-certs\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.991976 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-data\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992038 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec388004-374e-4c05-b088-c430a16326ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec388004-374e-4c05-b088-c430a16326ad-istio-podinfo\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992079 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-socket\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992183 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-kube-api-access-qlrn6\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-kube-api-access-qlrn6\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992228 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-certs" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:21.992488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992376 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-data" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:21.992488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992389 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec388004-374e-4c05-b088-c430a16326ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec388004-374e-4c05-b088-c430a16326ad-istiod-ca-cert\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992408 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-socket" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:21.992488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992447 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-credential-socket\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992482 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-istio-token\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-istio-token\") pod \"ec388004-374e-4c05-b088-c430a16326ad\" (UID: \"ec388004-374e-4c05-b088-c430a16326ad\") " Oct 31 22:13:21.992761 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992664 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-credential-socket" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:21.992761 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.992731 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec388004-374e-4c05-b088-c430a16326ad-istiod-ca-cert" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:21.993054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.993027 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec388004-374e-4c05-b088-c430a16326ad-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:21.993151 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.993055 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:21.993151 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.993071 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:21.993151 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.993083 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:21.993151 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.993095 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:21.994323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.994042 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-envoy" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:13:21.994423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.994340 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec388004-374e-4c05-b088-c430a16326ad-istio-podinfo" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:13:21.994423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.994415 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-kube-api-access-qlrn6" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "kube-api-access-qlrn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:21.994681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:21.994660 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-istio-token" pod "ec388004-374e-4c05-b088-c430a16326ad" (UID: "ec388004-374e-4c05-b088-c430a16326ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:22.094681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.094646 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec388004-374e-4c05-b088-c430a16326ad-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.094681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.094677 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlrn6\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-kube-api-access-qlrn6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.094681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.094688 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec388004-374e-4c05-b088-c430a16326ad-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.095206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.094698 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec388004-374e-4c05-b088-c430a16326ad-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:22.350060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.349983 4294 generic.go:356] "Generic (PLEG): container finished" podID="ec388004-374e-4c05-b088-c430a16326ad" containerID="90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1" exitCode=0 Oct 31 22:13:22.350207 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.350184 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" event={"ID":"ec388004-374e-4c05-b088-c430a16326ad","Type":"ContainerDied","Data":"90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1"} Oct 31 22:13:22.350254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.350236 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" event={"ID":"ec388004-374e-4c05-b088-c430a16326ad","Type":"ContainerDied","Data":"6a0249688b83039fc1e7911132383effc882c440ce307382fa426b42baaeb88c"} Oct 31 22:13:22.350294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.350267 4294 scope.go:122] "RemoveContainer" containerID="90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1" Oct 31 22:13:22.350463 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.350439 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f" Oct 31 22:13:22.387248 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.387223 4294 scope.go:122] "RemoveContainer" containerID="90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1" Oct 31 22:13:22.387666 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:13:22.387641 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1\": container with ID starting with 90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1 not found: ID does not exist" containerID="90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1" Oct 31 22:13:22.387752 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.387683 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1"} err="failed to get container status \"90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1\": rpc error: code = NotFound desc = could not find container \"90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1\": container with ID starting with 90a29236abc71e5ce4e5b20a45fe3e01771d36f2361d63fefe38cc375afa31d1 not found: ID does not exist" Oct 31 22:13:22.410895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.410859 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f"] Oct 31 22:13:22.422811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:22.422776 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4b5989-openshift-default-578774676c-7lx2f"] Oct 31 22:13:23.985972 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:23.985936 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec388004-374e-4c05-b088-c430a16326ad" path="/var/lib/kubelet/pods/ec388004-374e-4c05-b088-c430a16326ad/volumes" Oct 31 22:13:26.836311 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.836267 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k"] Oct 31 22:13:26.837162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.837143 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" Oct 31 22:13:26.837231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.837165 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" Oct 31 22:13:26.837231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.837183 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" containerName="maas-api" Oct 31 22:13:26.837231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.837189 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" containerName="maas-api" Oct 31 22:13:26.837374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.837325 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec388004-374e-4c05-b088-c430a16326ad" containerName="istio-proxy" Oct 31 22:13:26.837374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.837335 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1162f0-fbbf-4bfb-9bfd-e0977c2e40ea" containerName="maas-api" Oct 31 22:13:26.842643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.842623 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.859126 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.859100 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k"] Oct 31 22:13:26.865169 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.865143 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:13:26.865287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.865191 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:13:26.879002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.878975 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.879232 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.879207 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksq4p\" (UniqueName: \"kubernetes.io/projected/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-kube-api-access-ksq4p\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.879376 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.879357 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.981141 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.981110 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.981351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.981184 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.981351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.981288 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksq4p\" (UniqueName: \"kubernetes.io/projected/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-kube-api-access-ksq4p\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.981843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.981817 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:26.983589 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:26.983564 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:27.004351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:27.004290 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksq4p\" (UniqueName: \"kubernetes.io/projected/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-kube-api-access-ksq4p\") pod \"maas-api-revoke-sub-b3d8942b-hh79k\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:27.158683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:27.158616 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:27.462242 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:27.462197 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k"] Oct 31 22:13:27.462407 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:13:27.462262 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4eef1f7_82e5_4abf_b44e_8733b2d3a294.slice/crio-82d358027adb412919a5a659cb8d28979a426e65eba619c41f8417cf5483e695 WatchSource:0}: Error finding container 82d358027adb412919a5a659cb8d28979a426e65eba619c41f8417cf5483e695: Status 404 returned error can't find the container with id 82d358027adb412919a5a659cb8d28979a426e65eba619c41f8417cf5483e695 Oct 31 22:13:28.398015 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:28.397984 4294 generic.go:356] "Generic (PLEG): container finished" podID="e4eef1f7-82e5-4abf-b44e-8733b2d3a294" containerID="24938d4db348c162d76f2c4d2ba487cde3e183e71dadf7042c3799f7f3f4bf49" exitCode=0 Oct 31 22:13:28.398425 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:28.398041 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" event={"ID":"e4eef1f7-82e5-4abf-b44e-8733b2d3a294","Type":"ContainerDied","Data":"24938d4db348c162d76f2c4d2ba487cde3e183e71dadf7042c3799f7f3f4bf49"} Oct 31 22:13:28.398425 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:28.398088 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" event={"ID":"e4eef1f7-82e5-4abf-b44e-8733b2d3a294","Type":"ContainerStarted","Data":"82d358027adb412919a5a659cb8d28979a426e65eba619c41f8417cf5483e695"} Oct 31 22:13:29.578850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.578819 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:29.612162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.612132 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-xks-ca\") pod \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " Oct 31 22:13:29.612360 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.612200 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-kube-api-access-ksq4p\" (UniqueName: \"kubernetes.io/projected/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-kube-api-access-ksq4p\") pod \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " Oct 31 22:13:29.612360 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.612277 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-openshift-ca\") pod \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\" (UID: \"e4eef1f7-82e5-4abf-b44e-8733b2d3a294\") " Oct 31 22:13:29.612651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.612628 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-openshift-ca" pod "e4eef1f7-82e5-4abf-b44e-8733b2d3a294" (UID: "e4eef1f7-82e5-4abf-b44e-8733b2d3a294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:29.612935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.612918 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:29.614317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.614278 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-xks-ca" pod "e4eef1f7-82e5-4abf-b44e-8733b2d3a294" (UID: "e4eef1f7-82e5-4abf-b44e-8733b2d3a294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:13:29.614574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.614555 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-kube-api-access-ksq4p" pod "e4eef1f7-82e5-4abf-b44e-8733b2d3a294" (UID: "e4eef1f7-82e5-4abf-b44e-8733b2d3a294"). InnerVolumeSpecName "kube-api-access-ksq4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:29.714609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.714527 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:29.714609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:29.714566 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksq4p\" (UniqueName: \"kubernetes.io/projected/e4eef1f7-82e5-4abf-b44e-8733b2d3a294-kube-api-access-ksq4p\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:30.414557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:30.414484 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" Oct 31 22:13:30.414557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:30.414496 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k" event={"ID":"e4eef1f7-82e5-4abf-b44e-8733b2d3a294","Type":"ContainerDied","Data":"82d358027adb412919a5a659cb8d28979a426e65eba619c41f8417cf5483e695"} Oct 31 22:13:30.414557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:30.414542 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d358027adb412919a5a659cb8d28979a426e65eba619c41f8417cf5483e695" Oct 31 22:13:31.035732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:31.035700 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm"] Oct 31 22:13:31.039063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:31.039042 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e5b549e-4wjsm"] Oct 31 22:13:31.311384 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:13:31.311271 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:13:31.985526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:31.985497 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbf967bf-61f1-4176-8afb-ec496d753060" path="/var/lib/kubelet/pods/bbf967bf-61f1-4176-8afb-ec496d753060/volumes" Oct 31 22:13:37.734254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.734220 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss"] Oct 31 22:13:37.734864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.734848 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e4eef1f7-82e5-4abf-b44e-8733b2d3a294" containerName="revoke-keys" Oct 31 22:13:37.734916 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.734865 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4eef1f7-82e5-4abf-b44e-8733b2d3a294" containerName="revoke-keys" Oct 31 22:13:37.735012 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.735003 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4eef1f7-82e5-4abf-b44e-8733b2d3a294" containerName="revoke-keys" Oct 31 22:13:37.738466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.738450 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.755640 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.755609 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss"] Oct 31 22:13:37.763715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.763689 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:13:37.763861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.763757 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:13:37.831833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.831799 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed36186-d38a-4289-99ea-9fa780758130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.832017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.831930 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7ghj\" (UniqueName: \"kubernetes.io/projected/eed36186-d38a-4289-99ea-9fa780758130-kube-api-access-h7ghj\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.832141 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.832123 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed36186-d38a-4289-99ea-9fa780758130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.933896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.933864 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed36186-d38a-4289-99ea-9fa780758130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.934060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.934017 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed36186-d38a-4289-99ea-9fa780758130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.934131 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.934115 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7ghj\" (UniqueName: \"kubernetes.io/projected/eed36186-d38a-4289-99ea-9fa780758130-kube-api-access-h7ghj\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.934667 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.934640 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed36186-d38a-4289-99ea-9fa780758130-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.936055 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.936036 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed36186-d38a-4289-99ea-9fa780758130-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:37.959550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:37.959530 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7ghj\" (UniqueName: \"kubernetes.io/projected/eed36186-d38a-4289-99ea-9fa780758130-kube-api-access-h7ghj\") pod \"maas-api-revoke-sub-b7fab9f6-7jtss\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:38.052111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:38.052045 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:38.384435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:38.384411 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss"] Oct 31 22:13:38.385085 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:13:38.385059 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed36186_d38a_4289_99ea_9fa780758130.slice/crio-d319bcc9a63475ffd884cef58c2748b600942f4d38ecdf7bf9462067df19245c WatchSource:0}: Error finding container d319bcc9a63475ffd884cef58c2748b600942f4d38ecdf7bf9462067df19245c: Status 404 returned error can't find the container with id d319bcc9a63475ffd884cef58c2748b600942f4d38ecdf7bf9462067df19245c Oct 31 22:13:38.472218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:38.472184 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" event={"ID":"eed36186-d38a-4289-99ea-9fa780758130","Type":"ContainerStarted","Data":"d319bcc9a63475ffd884cef58c2748b600942f4d38ecdf7bf9462067df19245c"} Oct 31 22:13:39.480795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:39.480764 4294 generic.go:356] "Generic (PLEG): container finished" podID="eed36186-d38a-4289-99ea-9fa780758130" containerID="8e3a6ad107a85346d3184e4098dd943a4ca1b8448dcfa30ebf0beb008347f867" exitCode=0 Oct 31 22:13:39.481195 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:39.480848 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" event={"ID":"eed36186-d38a-4289-99ea-9fa780758130","Type":"ContainerDied","Data":"8e3a6ad107a85346d3184e4098dd943a4ca1b8448dcfa30ebf0beb008347f867"} Oct 31 22:13:40.634080 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.634055 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:40.769466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.769388 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed36186-d38a-4289-99ea-9fa780758130-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed36186-d38a-4289-99ea-9fa780758130-maas-api-openshift-ca\") pod \"eed36186-d38a-4289-99ea-9fa780758130\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " Oct 31 22:13:40.769642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.769516 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eed36186-d38a-4289-99ea-9fa780758130-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed36186-d38a-4289-99ea-9fa780758130-maas-api-xks-ca\") pod \"eed36186-d38a-4289-99ea-9fa780758130\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " Oct 31 22:13:40.769725 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.769667 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed36186-d38a-4289-99ea-9fa780758130-kube-api-access-h7ghj\" (UniqueName: \"kubernetes.io/projected/eed36186-d38a-4289-99ea-9fa780758130-kube-api-access-h7ghj\") pod \"eed36186-d38a-4289-99ea-9fa780758130\" (UID: \"eed36186-d38a-4289-99ea-9fa780758130\") " Oct 31 22:13:40.769804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.769775 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed36186-d38a-4289-99ea-9fa780758130-maas-api-openshift-ca" pod "eed36186-d38a-4289-99ea-9fa780758130" (UID: "eed36186-d38a-4289-99ea-9fa780758130"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:40.770149 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.770131 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed36186-d38a-4289-99ea-9fa780758130-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:40.771622 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.771601 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed36186-d38a-4289-99ea-9fa780758130-kube-api-access-h7ghj" pod "eed36186-d38a-4289-99ea-9fa780758130" (UID: "eed36186-d38a-4289-99ea-9fa780758130"). InnerVolumeSpecName "kube-api-access-h7ghj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:40.771694 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.771642 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed36186-d38a-4289-99ea-9fa780758130-maas-api-xks-ca" pod "eed36186-d38a-4289-99ea-9fa780758130" (UID: "eed36186-d38a-4289-99ea-9fa780758130"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:13:40.871909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.871874 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7ghj\" (UniqueName: \"kubernetes.io/projected/eed36186-d38a-4289-99ea-9fa780758130-kube-api-access-h7ghj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:40.871909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:40.871904 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed36186-d38a-4289-99ea-9fa780758130-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:41.497062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:41.497032 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" Oct 31 22:13:41.497062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:41.497052 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss" event={"ID":"eed36186-d38a-4289-99ea-9fa780758130","Type":"ContainerDied","Data":"d319bcc9a63475ffd884cef58c2748b600942f4d38ecdf7bf9462067df19245c"} Oct 31 22:13:41.497348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:41.497121 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d319bcc9a63475ffd884cef58c2748b600942f4d38ecdf7bf9462067df19245c" Oct 31 22:13:43.034250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:43.034173 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt"] Oct 31 22:13:43.060645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:43.060607 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3735e7a-wkgtt"] Oct 31 22:13:43.985979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:43.985946 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacb9938-e4e1-4e28-918e-0a6b26cdf014" path="/var/lib/kubelet/pods/aacb9938-e4e1-4e28-918e-0a6b26cdf014/volumes" Oct 31 22:13:49.531423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:49.531388 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww"] Oct 31 22:13:50.509525 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:50.509493 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx"] Oct 31 22:13:54.032717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.032686 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg"] Oct 31 22:13:54.044686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.044662 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc5756d5-zlggg"] Oct 31 22:13:54.531850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.531807 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" podUID="68bab017-a084-448b-b693-a0c9629329f5" containerName="payload-processing" containerID="cri-o://2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52" gracePeriod=25 Oct 31 22:13:54.826005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.825977 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:13:54.940207 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.940179 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-provider-credentials\") pod \"68bab017-a084-448b-b693-a0c9629329f5\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " Oct 31 22:13:54.940499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.940477 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-routing-overlay-volume\") pod \"68bab017-a084-448b-b693-a0c9629329f5\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " Oct 31 22:13:54.940579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.940534 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-kube-api-access-c7zzv\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-kube-api-access-c7zzv\") pod \"68bab017-a084-448b-b693-a0c9629329f5\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " Oct 31 22:13:54.940635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.940614 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-plugins-config-volume\") pod \"68bab017-a084-448b-b693-a0c9629329f5\" (UID: \"68bab017-a084-448b-b693-a0c9629329f5\") " Oct 31 22:13:54.940868 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.940844 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-routing-overlay-volume" pod "68bab017-a084-448b-b693-a0c9629329f5" (UID: "68bab017-a084-448b-b693-a0c9629329f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:54.941100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.941073 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-plugins-config-volume" pod "68bab017-a084-448b-b693-a0c9629329f5" (UID: "68bab017-a084-448b-b693-a0c9629329f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:54.941163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.941116 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:54.942414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.942398 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-provider-credentials" pod "68bab017-a084-448b-b693-a0c9629329f5" (UID: "68bab017-a084-448b-b693-a0c9629329f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:54.942528 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:54.942509 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-kube-api-access-c7zzv" pod "68bab017-a084-448b-b693-a0c9629329f5" (UID: "68bab017-a084-448b-b693-a0c9629329f5"). InnerVolumeSpecName "kube-api-access-c7zzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:55.042546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.042505 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68bab017-a084-448b-b693-a0c9629329f5-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:55.042546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.042543 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:55.043030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.042559 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7zzv\" (UniqueName: \"kubernetes.io/projected/68bab017-a084-448b-b693-a0c9629329f5-kube-api-access-c7zzv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:55.510615 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.510554 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" podUID="192f0920-4e5e-4727-9eaa-4c93744862fa" containerName="payload-processing" containerID="cri-o://743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e" gracePeriod=25 Oct 31 22:13:55.601730 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.601698 4294 generic.go:356] "Generic (PLEG): container finished" podID="68bab017-a084-448b-b693-a0c9629329f5" containerID="2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52" exitCode=0 Oct 31 22:13:55.601908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.601759 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" Oct 31 22:13:55.601908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.601775 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" event={"ID":"68bab017-a084-448b-b693-a0c9629329f5","Type":"ContainerDied","Data":"2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52"} Oct 31 22:13:55.601908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.601827 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww" event={"ID":"68bab017-a084-448b-b693-a0c9629329f5","Type":"ContainerDied","Data":"64adc76313754cb5e79e42573c1d3adb9316e108a8186368c761e832dfbd6ea7"} Oct 31 22:13:55.601908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.601848 4294 scope.go:122] "RemoveContainer" containerID="2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52" Oct 31 22:13:55.626369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.626346 4294 scope.go:122] "RemoveContainer" containerID="2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52" Oct 31 22:13:55.627735 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:13:55.627706 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52\": container with ID starting with 2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52 not found: ID does not exist" containerID="2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52" Oct 31 22:13:55.627872 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.627746 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52"} err="failed to get container status \"2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52\": rpc error: code = NotFound desc = could not find container \"2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52\": container with ID starting with 2b132ab75bd9a7227650ee8aa443daaae8295f84a73232b7d91917e04ee61e52 not found: ID does not exist" Oct 31 22:13:55.657530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.657413 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww"] Oct 31 22:13:55.671348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.671265 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-4275b9/payload-processing-external-model-e2e-extmod-a-4275b9-7f4b26vww"] Oct 31 22:13:55.960873 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.960840 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:13:55.998345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.998104 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68bab017-a084-448b-b693-a0c9629329f5" path="/var/lib/kubelet/pods/68bab017-a084-448b-b693-a0c9629329f5/volumes" Oct 31 22:13:55.999941 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:55.999911 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b220b73a-fc08-4891-b89e-4e78904b2510" path="/var/lib/kubelet/pods/b220b73a-fc08-4891-b89e-4e78904b2510/volumes" Oct 31 22:13:56.085111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.085012 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-plugins-config-volume\") pod \"192f0920-4e5e-4727-9eaa-4c93744862fa\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " Oct 31 22:13:56.085111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.085086 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-provider-credentials\") pod \"192f0920-4e5e-4727-9eaa-4c93744862fa\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " Oct 31 22:13:56.085710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.085115 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-routing-overlay-volume\") pod \"192f0920-4e5e-4727-9eaa-4c93744862fa\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " Oct 31 22:13:56.085710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.085173 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-kube-api-access-shwhj\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-kube-api-access-shwhj\") pod \"192f0920-4e5e-4727-9eaa-4c93744862fa\" (UID: \"192f0920-4e5e-4727-9eaa-4c93744862fa\") " Oct 31 22:13:56.102325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.088857 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-routing-overlay-volume" pod "192f0920-4e5e-4727-9eaa-4c93744862fa" (UID: "192f0920-4e5e-4727-9eaa-4c93744862fa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:56.102325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.089386 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-plugins-config-volume" pod "192f0920-4e5e-4727-9eaa-4c93744862fa" (UID: "192f0920-4e5e-4727-9eaa-4c93744862fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:13:56.161965 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.161920 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-provider-credentials" pod "192f0920-4e5e-4727-9eaa-4c93744862fa" (UID: "192f0920-4e5e-4727-9eaa-4c93744862fa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:56.183324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.162848 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-kube-api-access-shwhj" pod "192f0920-4e5e-4727-9eaa-4c93744862fa" (UID: "192f0920-4e5e-4727-9eaa-4c93744862fa"). InnerVolumeSpecName "kube-api-access-shwhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:13:56.204327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.199492 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:56.204327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.199528 4294 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-provider-credentials\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:56.204327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.199543 4294 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/192f0920-4e5e-4727-9eaa-4c93744862fa-routing-overlay-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:56.204327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.199557 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shwhj\" (UniqueName: \"kubernetes.io/projected/192f0920-4e5e-4727-9eaa-4c93744862fa-kube-api-access-shwhj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:13:56.658671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.658631 4294 generic.go:356] "Generic (PLEG): container finished" podID="192f0920-4e5e-4727-9eaa-4c93744862fa" containerID="743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e" exitCode=0 Oct 31 22:13:56.658870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.658732 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" event={"ID":"192f0920-4e5e-4727-9eaa-4c93744862fa","Type":"ContainerDied","Data":"743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e"} Oct 31 22:13:56.658870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.658793 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" event={"ID":"192f0920-4e5e-4727-9eaa-4c93744862fa","Type":"ContainerDied","Data":"6f5c5cd722549b63eefab4c31915a86adc4619e0280b1c74f9d5f93dcffeaca8"} Oct 31 22:13:56.658870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.658833 4294 scope.go:122] "RemoveContainer" containerID="743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e" Oct 31 22:13:56.659005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.658995 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx" Oct 31 22:13:56.679167 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.679138 4294 scope.go:122] "RemoveContainer" containerID="743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e" Oct 31 22:13:56.679677 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:13:56.679648 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e\": container with ID starting with 743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e not found: ID does not exist" containerID="743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e" Oct 31 22:13:56.679801 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.679688 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e"} err="failed to get container status \"743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e\": rpc error: code = NotFound desc = could not find container \"743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e\": container with ID starting with 743d5f049e403f9df10a8437b7fd79df229f8854c2d68ddcabe4b7603a30213e not found: ID does not exist" Oct 31 22:13:56.725361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.725331 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx"] Oct 31 22:13:56.731051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.731023 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a5affe/payload-processing-external-model-e2e-extmod-b-a5affe-68bbkl8rx"] Oct 31 22:13:56.868694 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.868659 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d"] Oct 31 22:13:56.869451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869430 4294 cpu_manager.go:414] "containerMap: removing container" podUID="192f0920-4e5e-4727-9eaa-4c93744862fa" containerName="payload-processing" Oct 31 22:13:56.869451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869451 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192f0920-4e5e-4727-9eaa-4c93744862fa" containerName="payload-processing" Oct 31 22:13:56.869629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869501 4294 cpu_manager.go:414] "containerMap: removing container" podUID="eed36186-d38a-4289-99ea-9fa780758130" containerName="revoke-keys" Oct 31 22:13:56.869629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869508 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed36186-d38a-4289-99ea-9fa780758130" containerName="revoke-keys" Oct 31 22:13:56.869629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869515 4294 cpu_manager.go:414] "containerMap: removing container" podUID="68bab017-a084-448b-b693-a0c9629329f5" containerName="payload-processing" Oct 31 22:13:56.869629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869520 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68bab017-a084-448b-b693-a0c9629329f5" containerName="payload-processing" Oct 31 22:13:56.869629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869626 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="192f0920-4e5e-4727-9eaa-4c93744862fa" containerName="payload-processing" Oct 31 22:13:56.869818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869637 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed36186-d38a-4289-99ea-9fa780758130" containerName="revoke-keys" Oct 31 22:13:56.869818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.869645 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="68bab017-a084-448b-b693-a0c9629329f5" containerName="payload-processing" Oct 31 22:13:56.874418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.874396 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:56.891451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.891423 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:13:56.891716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.891699 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:13:56.892620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:56.892599 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d"] Oct 31 22:13:57.014067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.014032 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.014233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.014187 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9g8f\" (UniqueName: \"kubernetes.io/projected/ba57bf51-444b-4228-8d83-7db04d104feb-kube-api-access-c9g8f\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.014280 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.014230 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.116164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.116130 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.116566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.116261 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9g8f\" (UniqueName: \"kubernetes.io/projected/ba57bf51-444b-4228-8d83-7db04d104feb-kube-api-access-c9g8f\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.116566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.116338 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.116873 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.116853 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.118800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.118778 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.139362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.139339 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9g8f\" (UniqueName: \"kubernetes.io/projected/ba57bf51-444b-4228-8d83-7db04d104feb-kube-api-access-c9g8f\") pod \"maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.188046 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.188020 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:13:57.491282 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.491251 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d"] Oct 31 22:13:57.491924 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:13:57.491893 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba57bf51_444b_4228_8d83_7db04d104feb.slice/crio-73ca8c755c161047fb6802bdd72738b487f308069f55712a1435cc9732fc670b WatchSource:0}: Error finding container 73ca8c755c161047fb6802bdd72738b487f308069f55712a1435cc9732fc670b: Status 404 returned error can't find the container with id 73ca8c755c161047fb6802bdd72738b487f308069f55712a1435cc9732fc670b Oct 31 22:13:57.680873 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.680842 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" event={"ID":"ba57bf51-444b-4228-8d83-7db04d104feb","Type":"ContainerStarted","Data":"73ca8c755c161047fb6802bdd72738b487f308069f55712a1435cc9732fc670b"} Oct 31 22:13:57.985384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:57.985351 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192f0920-4e5e-4727-9eaa-4c93744862fa" path="/var/lib/kubelet/pods/192f0920-4e5e-4727-9eaa-4c93744862fa/volumes" Oct 31 22:13:58.689913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:58.689833 4294 generic.go:356] "Generic (PLEG): container finished" podID="ba57bf51-444b-4228-8d83-7db04d104feb" containerID="ffa6a730e2e2a3e74e96fc52f223519a40175521a34ee494e14bf82dbacb4a86" exitCode=0 Oct 31 22:13:58.690267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:58.689920 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" event={"ID":"ba57bf51-444b-4228-8d83-7db04d104feb","Type":"ContainerDied","Data":"ffa6a730e2e2a3e74e96fc52f223519a40175521a34ee494e14bf82dbacb4a86"} Oct 31 22:13:59.889518 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:13:59.889487 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:14:00.057601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.057526 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba57bf51-444b-4228-8d83-7db04d104feb-kube-api-access-c9g8f\" (UniqueName: \"kubernetes.io/projected/ba57bf51-444b-4228-8d83-7db04d104feb-kube-api-access-c9g8f\") pod \"ba57bf51-444b-4228-8d83-7db04d104feb\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " Oct 31 22:14:00.057761 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.057729 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-xks-ca\") pod \"ba57bf51-444b-4228-8d83-7db04d104feb\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " Oct 31 22:14:00.057815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.057803 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-openshift-ca\") pod \"ba57bf51-444b-4228-8d83-7db04d104feb\" (UID: \"ba57bf51-444b-4228-8d83-7db04d104feb\") " Oct 31 22:14:00.058206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.058176 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-openshift-ca" pod "ba57bf51-444b-4228-8d83-7db04d104feb" (UID: "ba57bf51-444b-4228-8d83-7db04d104feb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:14:00.058376 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.058359 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:00.059687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.059664 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba57bf51-444b-4228-8d83-7db04d104feb-kube-api-access-c9g8f" pod "ba57bf51-444b-4228-8d83-7db04d104feb" (UID: "ba57bf51-444b-4228-8d83-7db04d104feb"). InnerVolumeSpecName "kube-api-access-c9g8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:14:00.059769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.059743 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-xks-ca" pod "ba57bf51-444b-4228-8d83-7db04d104feb" (UID: "ba57bf51-444b-4228-8d83-7db04d104feb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:14:00.160237 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.160199 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba57bf51-444b-4228-8d83-7db04d104feb-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:00.160237 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.160234 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9g8f\" (UniqueName: \"kubernetes.io/projected/ba57bf51-444b-4228-8d83-7db04d104feb-kube-api-access-c9g8f\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:00.741117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.741090 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" Oct 31 22:14:00.741268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.741124 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d" event={"ID":"ba57bf51-444b-4228-8d83-7db04d104feb","Type":"ContainerDied","Data":"73ca8c755c161047fb6802bdd72738b487f308069f55712a1435cc9732fc670b"} Oct 31 22:14:00.741268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:00.741170 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73ca8c755c161047fb6802bdd72738b487f308069f55712a1435cc9732fc670b" Oct 31 22:14:01.312232 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:14:01.312197 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:14:03.345805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:03.345774 4294 scope.go:122] "RemoveContainer" containerID="a32c4bbe856b4bad8197565a5869382476740110bd1a94cc30695396e2ca2030" Oct 31 22:14:06.033513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:06.033482 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl"] Oct 31 22:14:06.044863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:06.044841 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04d6e260-6btkl"] Oct 31 22:14:07.352868 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:07.352836 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh"] Oct 31 22:14:07.383556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:07.383526 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b"] Oct 31 22:14:07.985563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:07.985533 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e858c6-9c71-46c1-9fa4-e59a993604a8" path="/var/lib/kubelet/pods/71e858c6-9c71-46c1-9fa4-e59a993604a8/volumes" Oct 31 22:14:11.162349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.162318 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d"] Oct 31 22:14:11.163481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.163448 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ba57bf51-444b-4228-8d83-7db04d104feb" containerName="revoke-keys" Oct 31 22:14:11.163481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.163473 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba57bf51-444b-4228-8d83-7db04d104feb" containerName="revoke-keys" Oct 31 22:14:11.163670 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.163626 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba57bf51-444b-4228-8d83-7db04d104feb" containerName="revoke-keys" Oct 31 22:14:11.167589 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.167556 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.188367 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.188155 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d"] Oct 31 22:14:11.206697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.206663 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-cd421c-openshift-default-dockercfg-thr5c\"" Oct 31 22:14:11.207345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.207319 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-data\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.207511 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.207496 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988780d-ff11-46b9-b92d-19f654a6d224-istiod-ca-cert\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.207704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.207692 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-envoy\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.207842 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.207830 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988780d-ff11-46b9-b92d-19f654a6d224-istio-podinfo\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.207970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.207957 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-certs\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.208156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.208142 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-credential-socket\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.208267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.208256 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wht8\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-kube-api-access-7wht8\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.208453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.208439 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-socket\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.208593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.208581 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-istio-token\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.310324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.310204 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988780d-ff11-46b9-b92d-19f654a6d224-istiod-ca-cert\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.335162 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988780d-ff11-46b9-b92d-19f654a6d224-istiod-ca-cert\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.343514 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-envoy\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.358463 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988780d-ff11-46b9-b92d-19f654a6d224-istio-podinfo\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.358506 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-certs\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.358631 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-credential-socket\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.358652 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wht8\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-kube-api-access-7wht8\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.358766 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-socket\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.358826 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-istio-token\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.358889 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-data\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.359281 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-data\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.364220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.347493 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-envoy\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.367636 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.367605 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988780d-ff11-46b9-b92d-19f654a6d224-istio-podinfo\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.368407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.368383 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-certs\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.368844 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.368822 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-credential-socket\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.369202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.369184 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-socket\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.394535 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.394507 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wht8\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-kube-api-access-7wht8\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.397245 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.397199 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-istio-token\") pod \"e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.484402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.484372 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:11.805047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.805022 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d"] Oct 31 22:14:11.805852 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:14:11.805827 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4988780d_ff11_46b9_b92d_19f654a6d224.slice/crio-c74c85c81e48aacf2306d7192a26ad420eca7d36d18113608c1e4e2cfe8131ae WatchSource:0}: Error finding container c74c85c81e48aacf2306d7192a26ad420eca7d36d18113608c1e4e2cfe8131ae: Status 404 returned error can't find the container with id c74c85c81e48aacf2306d7192a26ad420eca7d36d18113608c1e4e2cfe8131ae Oct 31 22:14:11.809024 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.808991 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:14:11.809114 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.809057 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:14:11.809114 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.809088 4294 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 22:14:11.825994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:11.825967 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" event={"ID":"4988780d-ff11-46b9-b92d-19f654a6d224","Type":"ContainerStarted","Data":"c74c85c81e48aacf2306d7192a26ad420eca7d36d18113608c1e4e2cfe8131ae"} Oct 31 22:14:12.353927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:12.353882 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" podUID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" containerName="payload-processing" containerID="cri-o://ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6" gracePeriod=25 Oct 31 22:14:12.387327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:12.384374 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" podUID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" containerName="payload-pre-processing" containerID="cri-o://9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8" gracePeriod=25 Oct 31 22:14:12.835662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:12.835626 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" event={"ID":"4988780d-ff11-46b9-b92d-19f654a6d224","Type":"ContainerStarted","Data":"bc8fd27f1969e827304779455ca4241c0b70ea2723094dcb53bcdc9fd3a38971"} Oct 31 22:14:12.901773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:12.901721 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podStartSLOduration=1.9017065359999998 podStartE2EDuration="1.901706536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:14:12.897766759 +0000 UTC m=+3493.022873990" watchObservedRunningTime="2026-10-31 22:14:12.901706536 +0000 UTC m=+3493.026813831" Oct 31 22:14:13.486930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:13.486898 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:13.491321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:13.489388 4294 patch_prober.go:29] interesting pod/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" start-of-body= Oct 31 22:14:13.491321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:13.489461 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" Oct 31 22:14:14.496713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:14.496676 4294 patch_prober.go:29] interesting pod/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" start-of-body= Oct 31 22:14:14.497124 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:14.496744 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" Oct 31 22:14:15.491693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:15.486796 4294 patch_prober.go:29] interesting pod/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" start-of-body= Oct 31 22:14:15.491693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:15.486857 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" Oct 31 22:14:16.487707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:16.487667 4294 patch_prober.go:29] interesting pod/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" start-of-body= Oct 31 22:14:16.488266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:16.488235 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" Oct 31 22:14:17.487119 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:17.486658 4294 patch_prober.go:29] interesting pod/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" start-of-body= Oct 31 22:14:17.487119 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:17.486723 4294 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.62:15021/healthz/ready\": dial tcp 10.134.1.62:15021: connect: connection refused" Oct 31 22:14:18.017093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:18.017057 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" podUID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:14:18.017519 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:18.017071 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" podUID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:14:18.489426 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:18.489398 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:18.489803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:18.489783 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:18.490638 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:18.490617 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:14:21.368376 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.368341 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj"] Oct 31 22:14:21.372912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.372887 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.391192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.391163 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj"] Oct 31 22:14:21.398211 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.398186 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1b0cce\"/\"test-model-1b0cce-kserve-self-signed-certs\"" Oct 31 22:14:21.398211 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.398206 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1b0cce\"/\"openshift-service-ca.crt\"" Oct 31 22:14:21.398418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.398278 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1b0cce\"/\"kube-root-ca.crt\"" Oct 31 22:14:21.401497 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.401474 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1b0cce\"/\"default-dockercfg-mkcpl\"" Oct 31 22:14:21.433894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.433862 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-model-cache\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.434032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.433908 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s88gx\" (UniqueName: \"kubernetes.io/projected/bfde9ff9-1991-43e3-8e62-99c461f90862-kube-api-access-s88gx\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.434032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.434005 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfde9ff9-1991-43e3-8e62-99c461f90862-tls-certs\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.434184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.434169 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-dshm\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.434278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.434262 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-home\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.434413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.434395 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-tmp-dir\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536009 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.535969 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-model-cache\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536025 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s88gx\" (UniqueName: \"kubernetes.io/projected/bfde9ff9-1991-43e3-8e62-99c461f90862-kube-api-access-s88gx\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536051 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfde9ff9-1991-43e3-8e62-99c461f90862-tls-certs\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536154 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-dshm\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536428 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536222 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-home\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536428 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536333 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-tmp-dir\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536494 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-model-cache\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536712 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-home\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.536903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.536767 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-tmp-dir\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.538513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.538489 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-dshm\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.538801 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.538783 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfde9ff9-1991-43e3-8e62-99c461f90862-tls-certs\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.560181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.560142 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s88gx\" (UniqueName: \"kubernetes.io/projected/bfde9ff9-1991-43e3-8e62-99c461f90862-kube-api-access-s88gx\") pod \"test-model-1b0cce-kserve-76d5d847f9-p52lj\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:21.691913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:21.691815 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:22.024514 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:22.024487 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj"] Oct 31 22:14:22.026018 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:14:22.025991 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfde9ff9_1991_43e3_8e62_99c461f90862.slice/crio-f84aa22481d935ba5187dcf4c2d7761d6fcdf51d4201ea9e7a0565b91f930527 WatchSource:0}: Error finding container f84aa22481d935ba5187dcf4c2d7761d6fcdf51d4201ea9e7a0565b91f930527: Status 404 returned error can't find the container with id f84aa22481d935ba5187dcf4c2d7761d6fcdf51d4201ea9e7a0565b91f930527 Oct 31 22:14:22.929908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:22.929858 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" event={"ID":"bfde9ff9-1991-43e3-8e62-99c461f90862","Type":"ContainerStarted","Data":"db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f"} Oct 31 22:14:22.930369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:22.929922 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" event={"ID":"bfde9ff9-1991-43e3-8e62-99c461f90862","Type":"ContainerStarted","Data":"f84aa22481d935ba5187dcf4c2d7761d6fcdf51d4201ea9e7a0565b91f930527"} Oct 31 22:14:22.979604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:22.979546 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" podStartSLOduration=1.979532346 podStartE2EDuration="1.979532346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:14:22.976946268 +0000 UTC m=+3503.102053500" watchObservedRunningTime="2026-10-31 22:14:22.979532346 +0000 UTC m=+3503.104639575" Oct 31 22:14:23.693911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:23.693877 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:28.085470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:28.085420 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" podUID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:14:28.111364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:28.111272 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" podUID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:14:28.413748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:28.413666 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d"] Oct 31 22:14:28.425778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:28.425748 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4275b9-shp2d"] Oct 31 22:14:29.985494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:29.985457 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba57bf51-444b-4228-8d83-7db04d104feb" path="/var/lib/kubelet/pods/ba57bf51-444b-4228-8d83-7db04d104feb/volumes" Oct 31 22:14:31.312280 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:14:31.312239 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:14:31.693409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:31.693334 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:31.706153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:31.706124 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:32.022728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.022699 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:14:32.681672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.681643 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 22:14:32.708557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.708530 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 22:14:32.809881 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.809845 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-plugins-config-volume\") pod \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " Oct 31 22:14:32.810084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.809892 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-plugins-config-volume\") pod \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " Oct 31 22:14:32.810084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.809934 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-kube-api-access-kbmp4\" (UniqueName: \"kubernetes.io/projected/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-kube-api-access-kbmp4\") pod \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\" (UID: \"1fa8d876-0d59-4a3b-a51c-a6c695a88aea\") " Oct 31 22:14:32.810084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.810047 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-kube-api-access-zbsfd\" (UniqueName: \"kubernetes.io/projected/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-kube-api-access-zbsfd\") pod \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\" (UID: \"29aef6e9-a20a-4ffd-af1a-2ddb84ace701\") " Oct 31 22:14:32.810401 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.810370 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-plugins-config-volume" pod "1fa8d876-0d59-4a3b-a51c-a6c695a88aea" (UID: "1fa8d876-0d59-4a3b-a51c-a6c695a88aea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:14:32.810493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.810402 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-plugins-config-volume" pod "29aef6e9-a20a-4ffd-af1a-2ddb84ace701" (UID: "29aef6e9-a20a-4ffd-af1a-2ddb84ace701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:14:32.810666 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.810644 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:32.810727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.810672 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:32.812744 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.812723 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-kube-api-access-kbmp4" pod "1fa8d876-0d59-4a3b-a51c-a6c695a88aea" (UID: "1fa8d876-0d59-4a3b-a51c-a6c695a88aea"). InnerVolumeSpecName "kube-api-access-kbmp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:14:32.812833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.812772 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-kube-api-access-zbsfd" pod "29aef6e9-a20a-4ffd-af1a-2ddb84ace701" (UID: "29aef6e9-a20a-4ffd-af1a-2ddb84ace701"). InnerVolumeSpecName "kube-api-access-zbsfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:14:32.912115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.912042 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmp4\" (UniqueName: \"kubernetes.io/projected/1fa8d876-0d59-4a3b-a51c-a6c695a88aea-kube-api-access-kbmp4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:32.912115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:32.912069 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbsfd\" (UniqueName: \"kubernetes.io/projected/29aef6e9-a20a-4ffd-af1a-2ddb84ace701-kube-api-access-zbsfd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:33.020807 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.020762 4294 generic.go:356] "Generic (PLEG): container finished" podID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" containerID="ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6" exitCode=0 Oct 31 22:14:33.021017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.020811 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" Oct 31 22:14:33.021017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.020848 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" event={"ID":"29aef6e9-a20a-4ffd-af1a-2ddb84ace701","Type":"ContainerDied","Data":"ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6"} Oct 31 22:14:33.021017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.020911 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh" event={"ID":"29aef6e9-a20a-4ffd-af1a-2ddb84ace701","Type":"ContainerDied","Data":"034981e22417fc938d71a1997ef08f28a9bf369e9462805d7014fe4dfd6d312e"} Oct 31 22:14:33.021017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.020940 4294 scope.go:122] "RemoveContainer" containerID="ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6" Oct 31 22:14:33.022642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.022621 4294 generic.go:356] "Generic (PLEG): container finished" podID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" containerID="9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8" exitCode=0 Oct 31 22:14:33.022786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.022646 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" event={"ID":"1fa8d876-0d59-4a3b-a51c-a6c695a88aea","Type":"ContainerDied","Data":"9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8"} Oct 31 22:14:33.022786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.022675 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" Oct 31 22:14:33.022786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.022696 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b" event={"ID":"1fa8d876-0d59-4a3b-a51c-a6c695a88aea","Type":"ContainerDied","Data":"823c29957bcbcb9247099815b9b7f37e769b59e1f2a00fa813c614276b1e179c"} Oct 31 22:14:33.034120 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.034100 4294 scope.go:122] "RemoveContainer" containerID="ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6" Oct 31 22:14:33.034417 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:14:33.034399 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6\": container with ID starting with ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6 not found: ID does not exist" containerID="ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6" Oct 31 22:14:33.034477 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.034425 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6"} err="failed to get container status \"ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6\": rpc error: code = NotFound desc = could not find container \"ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6\": container with ID starting with ab28e9a1c872fb61802d371c9a8bdf541cd63f68b7b157b3a069102d810a39e6 not found: ID does not exist" Oct 31 22:14:33.034477 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.034443 4294 scope.go:122] "RemoveContainer" containerID="9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8" Oct 31 22:14:33.044379 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.044360 4294 scope.go:122] "RemoveContainer" containerID="9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8" Oct 31 22:14:33.044636 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:14:33.044617 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8\": container with ID starting with 9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8 not found: ID does not exist" containerID="9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8" Oct 31 22:14:33.044705 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.044642 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8"} err="failed to get container status \"9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8\": rpc error: code = NotFound desc = could not find container \"9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8\": container with ID starting with 9cb6627b463fde60b507e7a2a888dbfb2d39d79ecaebe8ec8a4b6d6e003a43e8 not found: ID does not exist" Oct 31 22:14:33.073229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.073191 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b"] Oct 31 22:14:33.079384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.079360 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-4275b9-5fb89685c5-w497b"] Oct 31 22:14:33.111876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.111837 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh"] Oct 31 22:14:33.116117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.116094 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-4275b9-67997cc74f-twxgh"] Oct 31 22:14:33.987596 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.987562 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" path="/var/lib/kubelet/pods/1fa8d876-0d59-4a3b-a51c-a6c695a88aea/volumes" Oct 31 22:14:33.988189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:33.988167 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" path="/var/lib/kubelet/pods/29aef6e9-a20a-4ffd-af1a-2ddb84ace701/volumes" Oct 31 22:14:36.139472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.139437 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k"] Oct 31 22:14:36.140074 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.140060 4294 cpu_manager.go:414] "containerMap: removing container" podUID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" containerName="payload-processing" Oct 31 22:14:36.140074 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.140074 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" containerName="payload-processing" Oct 31 22:14:36.140169 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.140086 4294 cpu_manager.go:414] "containerMap: removing container" podUID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" containerName="payload-pre-processing" Oct 31 22:14:36.140169 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.140091 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" containerName="payload-pre-processing" Oct 31 22:14:36.140254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.140181 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="29aef6e9-a20a-4ffd-af1a-2ddb84ace701" containerName="payload-processing" Oct 31 22:14:36.140254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.140190 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa8d876-0d59-4a3b-a51c-a6c695a88aea" containerName="payload-pre-processing" Oct 31 22:14:36.143880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.143857 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.161811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.161775 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k"] Oct 31 22:14:36.169390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.169356 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:14:36.169795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.169499 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:14:36.266038 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.266004 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp5p2\" (UniqueName: \"kubernetes.io/projected/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-kube-api-access-qp5p2\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.266224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.266050 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.266224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.266217 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.368202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.368163 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp5p2\" (UniqueName: \"kubernetes.io/projected/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-kube-api-access-qp5p2\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.368416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.368225 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.368496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.368440 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.369015 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.368992 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.371039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.371013 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.391747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.391674 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp5p2\" (UniqueName: \"kubernetes.io/projected/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-kube-api-access-qp5p2\") pod \"maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.459205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.459166 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:36.764462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:36.764435 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k"] Oct 31 22:14:36.765753 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:14:36.765721 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa46b9c_6693_49dc_a07d_15ecb0c46dbe.slice/crio-4fb8dab9a09dcaaed83014af008fdf7c738e86a22c263c057624189a4169064e WatchSource:0}: Error finding container 4fb8dab9a09dcaaed83014af008fdf7c738e86a22c263c057624189a4169064e: Status 404 returned error can't find the container with id 4fb8dab9a09dcaaed83014af008fdf7c738e86a22c263c057624189a4169064e Oct 31 22:14:37.062829 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:37.062740 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" event={"ID":"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe","Type":"ContainerStarted","Data":"4fb8dab9a09dcaaed83014af008fdf7c738e86a22c263c057624189a4169064e"} Oct 31 22:14:38.070832 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:38.070799 4294 generic.go:356] "Generic (PLEG): container finished" podID="9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" containerID="7f61500ccd811c1af5b7a7e091ffa658e6aac1b5cf8187ef6e3c1172ddca93e9" exitCode=0 Oct 31 22:14:38.071243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:38.070848 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" event={"ID":"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe","Type":"ContainerDied","Data":"7f61500ccd811c1af5b7a7e091ffa658e6aac1b5cf8187ef6e3c1172ddca93e9"} Oct 31 22:14:39.227653 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.227622 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:39.307377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.307344 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-kube-api-access-qp5p2\" (UniqueName: \"kubernetes.io/projected/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-kube-api-access-qp5p2\") pod \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " Oct 31 22:14:39.307581 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.307520 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-xks-ca\") pod \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " Oct 31 22:14:39.307707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.307690 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-openshift-ca\") pod \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\" (UID: \"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe\") " Oct 31 22:14:39.308139 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.308105 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-openshift-ca" pod "9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" (UID: "9aa46b9c-6693-49dc-a07d-15ecb0c46dbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:14:39.308513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.308489 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:39.310093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.310055 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-xks-ca" pod "9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" (UID: "9aa46b9c-6693-49dc-a07d-15ecb0c46dbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:14:39.310437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.310417 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-kube-api-access-qp5p2" pod "9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" (UID: "9aa46b9c-6693-49dc-a07d-15ecb0c46dbe"). InnerVolumeSpecName "kube-api-access-qp5p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:14:39.410373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.410262 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:39.410373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:39.410294 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp5p2\" (UniqueName: \"kubernetes.io/projected/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe-kube-api-access-qp5p2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:40.089382 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:40.089330 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" event={"ID":"9aa46b9c-6693-49dc-a07d-15ecb0c46dbe","Type":"ContainerDied","Data":"4fb8dab9a09dcaaed83014af008fdf7c738e86a22c263c057624189a4169064e"} Oct 31 22:14:40.089382 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:40.089350 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k" Oct 31 22:14:40.089606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:40.089391 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fb8dab9a09dcaaed83014af008fdf7c738e86a22c263c057624189a4169064e" Oct 31 22:14:43.653066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.652221 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs"] Oct 31 22:14:43.653599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.653160 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" containerName="revoke-keys" Oct 31 22:14:43.653599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.653183 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" containerName="revoke-keys" Oct 31 22:14:43.653599 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.653435 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" containerName="revoke-keys" Oct 31 22:14:43.658235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.658213 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.683651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.683620 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs"] Oct 31 22:14:43.688756 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.688685 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5c54fc\"/\"test-model-5c54fc-kserve-self-signed-certs\"" Oct 31 22:14:43.688927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.688798 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5c54fc\"/\"openshift-service-ca.crt\"" Oct 31 22:14:43.688927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.688891 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5c54fc\"/\"default-dockercfg-hpdgf\"" Oct 31 22:14:43.689043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.688966 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5c54fc\"/\"kube-root-ca.crt\"" Oct 31 22:14:43.779949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.779911 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-dshm\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.780142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.779960 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-home\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.780142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.779986 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff86642-c360-4e03-a883-11efd7d6fdfe-tls-certs\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.780235 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.780165 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-model-cache\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.780390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.780365 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4swqx\" (UniqueName: \"kubernetes.io/projected/cff86642-c360-4e03-a883-11efd7d6fdfe-kube-api-access-4swqx\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.780538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.780523 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-tmp-dir\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.882129 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.882093 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4swqx\" (UniqueName: \"kubernetes.io/projected/cff86642-c360-4e03-a883-11efd7d6fdfe-kube-api-access-4swqx\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.882344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.882215 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-tmp-dir\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.882438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.882411 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-dshm\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.882827 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.882461 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-home\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.882827 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.882489 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff86642-c360-4e03-a883-11efd7d6fdfe-tls-certs\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.882827 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.882552 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-model-cache\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.883025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.882960 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-model-cache\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.883213 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.883188 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-home\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.883323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.883246 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-tmp-dir\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.885551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.885525 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-dshm\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.885683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.885579 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff86642-c360-4e03-a883-11efd7d6fdfe-tls-certs\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.916327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.916059 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4swqx\" (UniqueName: \"kubernetes.io/projected/cff86642-c360-4e03-a883-11efd7d6fdfe-kube-api-access-4swqx\") pod \"test-model-5c54fc-kserve-bd9466695-vkdcs\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:43.994616 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:43.994571 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:44.362990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:44.362955 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs"] Oct 31 22:14:44.365049 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:14:44.365022 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcff86642_c360_4e03_a883_11efd7d6fdfe.slice/crio-0199c863d64bf5c306458ff2303d15e25b089c9d2db78405d6e845a089f5b6ec WatchSource:0}: Error finding container 0199c863d64bf5c306458ff2303d15e25b089c9d2db78405d6e845a089f5b6ec: Status 404 returned error can't find the container with id 0199c863d64bf5c306458ff2303d15e25b089c9d2db78405d6e845a089f5b6ec Oct 31 22:14:45.140474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:45.140431 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" event={"ID":"cff86642-c360-4e03-a883-11efd7d6fdfe","Type":"ContainerStarted","Data":"f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558"} Oct 31 22:14:45.140876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:45.140489 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" event={"ID":"cff86642-c360-4e03-a883-11efd7d6fdfe","Type":"ContainerStarted","Data":"0199c863d64bf5c306458ff2303d15e25b089c9d2db78405d6e845a089f5b6ec"} Oct 31 22:14:45.184803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:45.184724 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" podStartSLOduration=2.184704329 podStartE2EDuration="2.184704329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:14:45.18171015 +0000 UTC m=+3525.306817374" watchObservedRunningTime="2026-10-31 22:14:45.184704329 +0000 UTC m=+3525.309811556" Oct 31 22:14:45.997191 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:45.997160 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:49.486482 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.486447 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq"] Oct 31 22:14:49.490818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.490792 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.508840 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.508811 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq"] Oct 31 22:14:49.515440 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.515419 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:14:49.515566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.515462 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:14:49.586253 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.586225 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.586420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.586282 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.586420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.586367 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmll\" (UniqueName: \"kubernetes.io/projected/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-kube-api-access-jdmll\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.687815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.687789 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.687925 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.687845 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.687925 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.687869 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmll\" (UniqueName: \"kubernetes.io/projected/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-kube-api-access-jdmll\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.688524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.688506 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.690073 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.690052 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.710920 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.710899 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmll\" (UniqueName: \"kubernetes.io/projected/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-kube-api-access-jdmll\") pod \"maas-api-revoke-sub-2ef0d0e1-7hqjq\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:49.806052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:49.805991 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:50.114052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:50.113947 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq"] Oct 31 22:14:50.121626 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:14:50.121594 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf1384d_e0b7_46cc_973a_07d7f4231b4d.slice/crio-ffd68f14cfaecb568b3da16ff9f65692eec2e5fd3c2cf88e13e5c96d7173f491 WatchSource:0}: Error finding container ffd68f14cfaecb568b3da16ff9f65692eec2e5fd3c2cf88e13e5c96d7173f491: Status 404 returned error can't find the container with id ffd68f14cfaecb568b3da16ff9f65692eec2e5fd3c2cf88e13e5c96d7173f491 Oct 31 22:14:50.177561 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:50.177537 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" event={"ID":"aaf1384d-e0b7-46cc-973a-07d7f4231b4d","Type":"ContainerStarted","Data":"ffd68f14cfaecb568b3da16ff9f65692eec2e5fd3c2cf88e13e5c96d7173f491"} Oct 31 22:14:51.186579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:51.186480 4294 generic.go:356] "Generic (PLEG): container finished" podID="aaf1384d-e0b7-46cc-973a-07d7f4231b4d" containerID="725976b53ee7e10345c21db4d5b82835184c029b52158d4f1b71f58a55d74f6c" exitCode=0 Oct 31 22:14:51.186579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:51.186563 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" event={"ID":"aaf1384d-e0b7-46cc-973a-07d7f4231b4d","Type":"ContainerDied","Data":"725976b53ee7e10345c21db4d5b82835184c029b52158d4f1b71f58a55d74f6c"} Oct 31 22:14:52.341282 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.341259 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:52.425467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.425439 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-openshift-ca\") pod \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " Oct 31 22:14:52.425610 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.425552 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-xks-ca\") pod \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " Oct 31 22:14:52.425650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.425621 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-kube-api-access-jdmll\" (UniqueName: \"kubernetes.io/projected/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-kube-api-access-jdmll\") pod \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\" (UID: \"aaf1384d-e0b7-46cc-973a-07d7f4231b4d\") " Oct 31 22:14:52.425779 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.425757 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-openshift-ca" pod "aaf1384d-e0b7-46cc-973a-07d7f4231b4d" (UID: "aaf1384d-e0b7-46cc-973a-07d7f4231b4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:14:52.426115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.426093 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:52.427699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.427672 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-xks-ca" pod "aaf1384d-e0b7-46cc-973a-07d7f4231b4d" (UID: "aaf1384d-e0b7-46cc-973a-07d7f4231b4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:14:52.427699 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.427692 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-kube-api-access-jdmll" pod "aaf1384d-e0b7-46cc-973a-07d7f4231b4d" (UID: "aaf1384d-e0b7-46cc-973a-07d7f4231b4d"). InnerVolumeSpecName "kube-api-access-jdmll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:14:52.526956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.526931 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:52.526956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:52.526955 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdmll\" (UniqueName: \"kubernetes.io/projected/aaf1384d-e0b7-46cc-973a-07d7f4231b4d-kube-api-access-jdmll\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:14:53.203662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:53.203623 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" event={"ID":"aaf1384d-e0b7-46cc-973a-07d7f4231b4d","Type":"ContainerDied","Data":"ffd68f14cfaecb568b3da16ff9f65692eec2e5fd3c2cf88e13e5c96d7173f491"} Oct 31 22:14:53.203857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:53.203684 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd68f14cfaecb568b3da16ff9f65692eec2e5fd3c2cf88e13e5c96d7173f491" Oct 31 22:14:53.203857 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:53.203637 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq" Oct 31 22:14:53.997731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:53.997703 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:54.009751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:54.009726 4294 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:14:54.221740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:14:54.221709 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:15:00.039180 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.039143 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs"] Oct 31 22:15:00.048200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.048172 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18a39add-gmmcs"] Oct 31 22:15:00.142376 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.142345 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc"] Oct 31 22:15:00.142995 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.142979 4294 cpu_manager.go:414] "containerMap: removing container" podUID="aaf1384d-e0b7-46cc-973a-07d7f4231b4d" containerName="revoke-keys" Oct 31 22:15:00.143061 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.142998 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf1384d-e0b7-46cc-973a-07d7f4231b4d" containerName="revoke-keys" Oct 31 22:15:00.143188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.143175 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf1384d-e0b7-46cc-973a-07d7f4231b4d" containerName="revoke-keys" Oct 31 22:15:00.148081 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.148061 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms"] Oct 31 22:15:00.148883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.148861 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.152562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.152542 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.153535 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.152792 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm"] Oct 31 22:15:00.157803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.157783 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp"] Oct 31 22:15:00.157962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.157894 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.192320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.191778 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:15:00.197421 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.193464 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn"] Oct 31 22:15:00.197421 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.195998 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.204153 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.204129 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.207472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.207452 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc"] Oct 31 22:15:00.222647 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.222626 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm"] Oct 31 22:15:00.229347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.228896 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn"] Oct 31 22:15:00.242349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.237644 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms"] Oct 31 22:15:00.242349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.237687 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4"] Oct 31 22:15:00.242349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.240786 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.242776 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.242757 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp"] Oct 31 22:15:00.242973 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.242956 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.248096 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.248075 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn"] Oct 31 22:15:00.255840 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.255815 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt"] Oct 31 22:15:00.277556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.277534 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd"] Oct 31 22:15:00.277911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.277744 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.285545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.285524 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g"] Oct 31 22:15:00.285929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.285911 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.295998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.295532 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn"] Oct 31 22:15:00.295998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.295575 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4"] Oct 31 22:15:00.295998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.295602 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv"] Oct 31 22:15:00.295998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.295637 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.300480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.300445 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz"] Oct 31 22:15:00.303414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.302253 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.305516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.305494 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt"] Oct 31 22:15:00.305619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.305531 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g"] Oct 31 22:15:00.305619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.305556 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.308525 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.308509 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd"] Oct 31 22:15:00.312480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.312312 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw"] Oct 31 22:15:00.318466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.318447 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt"] Oct 31 22:15:00.319231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.319214 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.324412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.324392 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz"] Oct 31 22:15:00.324530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.324468 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:00.332878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.332377 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv"] Oct 31 22:15:00.334137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334109 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn9vx\" (UniqueName: \"kubernetes.io/projected/8db9dfad-1040-4e7c-8203-1497b21954f0-kube-api-access-hn9vx\") pod \"maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.334246 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334160 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc62938-ef10-4235-91af-d518a045ce95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.334246 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334198 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.334434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334267 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5td\" (UniqueName: \"kubernetes.io/projected/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-kube-api-access-qz5td\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.334434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334341 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hjp\" (UniqueName: \"kubernetes.io/projected/3ee08211-d158-43a4-a055-61d92d2ee655-kube-api-access-b5hjp\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.334434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334365 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxd7\" (UniqueName: \"kubernetes.io/projected/f6e5c17f-0945-4321-b1d4-26c06bfd778c-kube-api-access-6jxd7\") pod \"maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.334434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334391 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee08211-d158-43a4-a055-61d92d2ee655-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.334434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334426 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmn9\" (UniqueName: \"kubernetes.io/projected/728fa16a-ae89-4220-8990-7ac122157f07-kube-api-access-hsmn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.334767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334500 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845f9969-a52b-4d80-a641-81c2e8020c54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.334767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334537 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/845f9969-a52b-4d80-a641-81c2e8020c54-kube-api-access-z5mms\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.334767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334600 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728fa16a-ae89-4220-8990-7ac122157f07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.334767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334656 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e5c17f-0945-4321-b1d4-26c06bfd778c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.334767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334731 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxt4k\" (UniqueName: \"kubernetes.io/projected/4bc62938-ef10-4235-91af-d518a045ce95-kube-api-access-zxt4k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.335057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.334808 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db9dfad-1040-4e7c-8203-1497b21954f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.336785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.336762 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw"] Oct 31 22:15:00.344993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.344973 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt"] Oct 31 22:15:00.436012 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.435980 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56php\" (UniqueName: \"kubernetes.io/projected/d7a1dfb1-a6f3-42fa-8566-bb614078045a-kube-api-access-56php\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.436238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436031 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxt4k\" (UniqueName: \"kubernetes.io/projected/4bc62938-ef10-4235-91af-d518a045ce95-kube-api-access-zxt4k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.436238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436056 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c519470-3a10-488d-a542-20423d52211a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.436238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436191 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.436445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436243 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzgfw\" (UniqueName: \"kubernetes.io/projected/9f0741c6-c011-4c39-b014-0d9034978b03-kube-api-access-vzgfw\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.436445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436287 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqvq\" (UniqueName: \"kubernetes.io/projected/3aa3eb2b-4e44-4796-904f-a9aec7047555-kube-api-access-hqqvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.436445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436340 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db9dfad-1040-4e7c-8203-1497b21954f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.436445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436398 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gdcz\" (UniqueName: \"kubernetes.io/projected/31db38fe-c0de-4701-b7ed-62b8a7539a1a-kube-api-access-6gdcz\") pod \"maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.436445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436443 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn9vx\" (UniqueName: \"kubernetes.io/projected/8db9dfad-1040-4e7c-8203-1497b21954f0-kube-api-access-hn9vx\") pod \"maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.436721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436493 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc62938-ef10-4235-91af-d518a045ce95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.436721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436517 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.436721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436551 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj5ml\" (UniqueName: \"kubernetes.io/projected/4312b8c4-a53c-4354-b696-695e71f214d6-kube-api-access-vj5ml\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:00.436721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436612 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a1dfb1-a6f3-42fa-8566-bb614078045a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.436721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436666 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0741c6-c011-4c39-b014-0d9034978b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.436721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436692 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5td\" (UniqueName: \"kubernetes.io/projected/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-kube-api-access-qz5td\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.437075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436731 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4312b8c4-a53c-4354-b696-695e71f214d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:00.437075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436765 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31db38fe-c0de-4701-b7ed-62b8a7539a1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.437075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436799 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hjp\" (UniqueName: \"kubernetes.io/projected/3ee08211-d158-43a4-a055-61d92d2ee655-kube-api-access-b5hjp\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.437075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436824 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxd7\" (UniqueName: \"kubernetes.io/projected/f6e5c17f-0945-4321-b1d4-26c06bfd778c-kube-api-access-6jxd7\") pod \"maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.437075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436881 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee08211-d158-43a4-a055-61d92d2ee655-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.437075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436921 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3eb2b-4e44-4796-904f-a9aec7047555-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.437075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.436969 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmn9\" (UniqueName: \"kubernetes.io/projected/728fa16a-ae89-4220-8990-7ac122157f07-kube-api-access-hsmn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.437389 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.437365 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db9dfad-1040-4e7c-8203-1497b21954f0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.437434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.437388 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc62938-ef10-4235-91af-d518a045ce95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.438964 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.438942 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.439328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.439285 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845f9969-a52b-4d80-a641-81c2e8020c54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.439377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.439344 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/845f9969-a52b-4d80-a641-81c2e8020c54-kube-api-access-z5mms\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.439417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.439373 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vkd\" (UniqueName: \"kubernetes.io/projected/7c519470-3a10-488d-a542-20423d52211a-kube-api-access-49vkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.439454 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.439428 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8rk4\" (UniqueName: \"kubernetes.io/projected/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-kube-api-access-k8rk4\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.439492 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.439464 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728fa16a-ae89-4220-8990-7ac122157f07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.439527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.439512 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee08211-d158-43a4-a055-61d92d2ee655-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.439562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.439522 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e5c17f-0945-4321-b1d4-26c06bfd778c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.440158 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.440127 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e5c17f-0945-4321-b1d4-26c06bfd778c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.440311 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.440283 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845f9969-a52b-4d80-a641-81c2e8020c54-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.440801 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.440775 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728fa16a-ae89-4220-8990-7ac122157f07-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.454352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.454323 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp"] Oct 31 22:15:00.471396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.471368 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hjp\" (UniqueName: \"kubernetes.io/projected/3ee08211-d158-43a4-a055-61d92d2ee655-kube-api-access-b5hjp\") pod \"maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.471537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.471421 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn9vx\" (UniqueName: \"kubernetes.io/projected/8db9dfad-1040-4e7c-8203-1497b21954f0-kube-api-access-hn9vx\") pod \"maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.472003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.471974 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5td\" (UniqueName: \"kubernetes.io/projected/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-kube-api-access-qz5td\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.473861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.473832 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxd7\" (UniqueName: \"kubernetes.io/projected/f6e5c17f-0945-4321-b1d4-26c06bfd778c-kube-api-access-6jxd7\") pod \"maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.474292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.474266 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmn9\" (UniqueName: \"kubernetes.io/projected/728fa16a-ae89-4220-8990-7ac122157f07-kube-api-access-hsmn9\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.475822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.475748 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/845f9969-a52b-4d80-a641-81c2e8020c54-kube-api-access-z5mms\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.483847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.483826 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxt4k\" (UniqueName: \"kubernetes.io/projected/4bc62938-ef10-4235-91af-d518a045ce95-kube-api-access-zxt4k\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.503256 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.503237 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:00.524953 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.524927 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:00.541592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541557 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4312b8c4-a53c-4354-b696-695e71f214d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:00.541733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541613 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31db38fe-c0de-4701-b7ed-62b8a7539a1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.541733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541664 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3eb2b-4e44-4796-904f-a9aec7047555-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.541940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541750 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vkd\" (UniqueName: \"kubernetes.io/projected/7c519470-3a10-488d-a542-20423d52211a-kube-api-access-49vkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.541940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541803 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8rk4\" (UniqueName: \"kubernetes.io/projected/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-kube-api-access-k8rk4\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.541940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541903 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56php\" (UniqueName: \"kubernetes.io/projected/d7a1dfb1-a6f3-42fa-8566-bb614078045a-kube-api-access-56php\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.542102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541938 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c519470-3a10-488d-a542-20423d52211a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.542102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.541974 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.542102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.542013 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzgfw\" (UniqueName: \"kubernetes.io/projected/9f0741c6-c011-4c39-b014-0d9034978b03-kube-api-access-vzgfw\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.542102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.542069 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqvq\" (UniqueName: \"kubernetes.io/projected/3aa3eb2b-4e44-4796-904f-a9aec7047555-kube-api-access-hqqvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.542494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.542123 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gdcz\" (UniqueName: \"kubernetes.io/projected/31db38fe-c0de-4701-b7ed-62b8a7539a1a-kube-api-access-6gdcz\") pod \"maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.542494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.542211 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj5ml\" (UniqueName: \"kubernetes.io/projected/4312b8c4-a53c-4354-b696-695e71f214d6-kube-api-access-vj5ml\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:00.542494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.542254 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a1dfb1-a6f3-42fa-8566-bb614078045a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.542494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.542278 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31db38fe-c0de-4701-b7ed-62b8a7539a1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.543381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.543354 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0741c6-c011-4c39-b014-0d9034978b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.543500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.543458 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.544130 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.544105 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a1dfb1-a6f3-42fa-8566-bb614078045a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.544256 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.544186 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0741c6-c011-4c39-b014-0d9034978b03-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.545430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.545408 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c519470-3a10-488d-a542-20423d52211a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.552042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.548201 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3eb2b-4e44-4796-904f-a9aec7047555-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.571975 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.571943 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4312b8c4-a53c-4354-b696-695e71f214d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:00.583910 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.583425 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vkd\" (UniqueName: \"kubernetes.io/projected/7c519470-3a10-488d-a542-20423d52211a-kube-api-access-49vkd\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.591174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.591144 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8rk4\" (UniqueName: \"kubernetes.io/projected/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-kube-api-access-k8rk4\") pod \"maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.591613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.591591 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:00.594740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.594086 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzgfw\" (UniqueName: \"kubernetes.io/projected/9f0741c6-c011-4c39-b014-0d9034978b03-kube-api-access-vzgfw\") pod \"maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.594740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.594561 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:00.594740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.594645 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:00.594740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.594710 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:00.631387 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.631024 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:00.631387 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.631230 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:00.646941 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.645551 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:00.668739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.668672 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:00.681437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.681414 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqvq\" (UniqueName: \"kubernetes.io/projected/3aa3eb2b-4e44-4796-904f-a9aec7047555-kube-api-access-hqqvq\") pod \"maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.681623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.681590 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj5ml\" (UniqueName: \"kubernetes.io/projected/4312b8c4-a53c-4354-b696-695e71f214d6-kube-api-access-vj5ml\") pod \"maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:00.681696 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.681471 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56php\" (UniqueName: \"kubernetes.io/projected/d7a1dfb1-a6f3-42fa-8566-bb614078045a-kube-api-access-56php\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.684040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.684000 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gdcz\" (UniqueName: \"kubernetes.io/projected/31db38fe-c0de-4701-b7ed-62b8a7539a1a-kube-api-access-6gdcz\") pod \"maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.952570 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.936730 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:00.955023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.954702 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:00.965640 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.965368 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc"] Oct 31 22:15:00.969854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.968547 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:00.981521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:00.978421 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:01.235603 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.234523 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee08211_d158_43a4_a055_61d92d2ee655.slice/crio-35f95a02bb47eb910d1493bf68822487d1f57c7117f52255e5159b315fb09bdb WatchSource:0}: Error finding container 35f95a02bb47eb910d1493bf68822487d1f57c7117f52255e5159b315fb09bdb: Status 404 returned error can't find the container with id 35f95a02bb47eb910d1493bf68822487d1f57c7117f52255e5159b315fb09bdb Oct 31 22:15:01.235603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.234756 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms"] Oct 31 22:15:01.257618 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.257594 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp"] Oct 31 22:15:01.258568 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.258428 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e5c17f_0945_4321_b1d4_26c06bfd778c.slice/crio-6acbedb3b21585dbdf7fb2a12078b291c22f6f7e3e0a0d8437512797eae4bb86 WatchSource:0}: Error finding container 6acbedb3b21585dbdf7fb2a12078b291c22f6f7e3e0a0d8437512797eae4bb86: Status 404 returned error can't find the container with id 6acbedb3b21585dbdf7fb2a12078b291c22f6f7e3e0a0d8437512797eae4bb86 Oct 31 22:15:01.271526 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.271502 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" event={"ID":"f6e5c17f-0945-4321-b1d4-26c06bfd778c","Type":"ContainerStarted","Data":"6acbedb3b21585dbdf7fb2a12078b291c22f6f7e3e0a0d8437512797eae4bb86"} Oct 31 22:15:01.272747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.272718 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" event={"ID":"3ee08211-d158-43a4-a055-61d92d2ee655","Type":"ContainerStarted","Data":"35f95a02bb47eb910d1493bf68822487d1f57c7117f52255e5159b315fb09bdb"} Oct 31 22:15:01.274112 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.274089 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" event={"ID":"a541b3ad-0cf9-4a2e-b488-f2d54d250c83","Type":"ContainerStarted","Data":"462da6e4acbe1f3204662d78b1a8123e1b4f8f6e3ef9566d2c56f0b3ca132ab6"} Oct 31 22:15:01.329429 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.329393 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn"] Oct 31 22:15:01.362184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.362125 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw"] Oct 31 22:15:01.374269 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.374244 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn"] Oct 31 22:15:01.393527 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:01.393495 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:15:01.393632 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.393595 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt"] Oct 31 22:15:01.396760 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.396736 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm"] Oct 31 22:15:01.410366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.410345 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g"] Oct 31 22:15:01.418559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.418540 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt"] Oct 31 22:15:01.444219 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.444198 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4"] Oct 31 22:15:01.479429 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.479403 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod728fa16a_ae89_4220_8990_7ac122157f07.slice/crio-5273aa262c0a4e5c13b26c802a8baf546e0c4b162bdd79f8ce4fd89f58562ebb WatchSource:0}: Error finding container 5273aa262c0a4e5c13b26c802a8baf546e0c4b162bdd79f8ce4fd89f58562ebb: Status 404 returned error can't find the container with id 5273aa262c0a4e5c13b26c802a8baf546e0c4b162bdd79f8ce4fd89f58562ebb Oct 31 22:15:01.481359 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.481336 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a772d8_9d68_4a2f_a241_6f356eb5a2c1.slice/crio-537efebfe3a05725212fc2f09fe1c9d6dfd3af088ed3f5288413c60564750a5c WatchSource:0}: Error finding container 537efebfe3a05725212fc2f09fe1c9d6dfd3af088ed3f5288413c60564750a5c: Status 404 returned error can't find the container with id 537efebfe3a05725212fc2f09fe1c9d6dfd3af088ed3f5288413c60564750a5c Oct 31 22:15:01.484058 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.484038 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c519470_3a10_488d_a542_20423d52211a.slice/crio-87aa8775bac7a50f5c663a731d7e9e1290984127a0591c1910a10336e869469c WatchSource:0}: Error finding container 87aa8775bac7a50f5c663a731d7e9e1290984127a0591c1910a10336e869469c: Status 404 returned error can't find the container with id 87aa8775bac7a50f5c663a731d7e9e1290984127a0591c1910a10336e869469c Oct 31 22:15:01.485121 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.485071 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845f9969_a52b_4d80_a641_81c2e8020c54.slice/crio-e7d8d8e2269f1a4e0c04db8a676084764c3ed6b739d8e1e13f275b9691a43f2f WatchSource:0}: Error finding container e7d8d8e2269f1a4e0c04db8a676084764c3ed6b739d8e1e13f275b9691a43f2f: Status 404 returned error can't find the container with id e7d8d8e2269f1a4e0c04db8a676084764c3ed6b739d8e1e13f275b9691a43f2f Oct 31 22:15:01.526985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.526916 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd"] Oct 31 22:15:01.536956 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.536921 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7a1dfb1_a6f3_42fa_8566_bb614078045a.slice/crio-b0a8d5e81705252b36fdaba40393c29a63375448b5da614967ada26e505cb9bb WatchSource:0}: Error finding container b0a8d5e81705252b36fdaba40393c29a63375448b5da614967ada26e505cb9bb: Status 404 returned error can't find the container with id b0a8d5e81705252b36fdaba40393c29a63375448b5da614967ada26e505cb9bb Oct 31 22:15:01.563211 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.563188 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv"] Oct 31 22:15:01.569976 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.569950 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz"] Oct 31 22:15:01.570073 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.570010 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa3eb2b_4e44_4796_904f_a9aec7047555.slice/crio-8b3fe72a6ed98d74b3e0925ba9597dc6d9d5fdc0d6c24ab6024cf928ac6111a9 WatchSource:0}: Error finding container 8b3fe72a6ed98d74b3e0925ba9597dc6d9d5fdc0d6c24ab6024cf928ac6111a9: Status 404 returned error can't find the container with id 8b3fe72a6ed98d74b3e0925ba9597dc6d9d5fdc0d6c24ab6024cf928ac6111a9 Oct 31 22:15:01.573629 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:01.573490 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31db38fe_c0de_4701_b7ed_62b8a7539a1a.slice/crio-cc08b4d7149ff4676269e8dcf01578f124f15a08b64b571a39eff598488674ab WatchSource:0}: Error finding container cc08b4d7149ff4676269e8dcf01578f124f15a08b64b571a39eff598488674ab: Status 404 returned error can't find the container with id cc08b4d7149ff4676269e8dcf01578f124f15a08b64b571a39eff598488674ab Oct 31 22:15:01.989830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:01.989792 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7390b4-4c4d-4230-b496-374f12ab0736" path="/var/lib/kubelet/pods/0b7390b4-4c4d-4230-b496-374f12ab0736/volumes" Oct 31 22:15:02.296203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.296173 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerStarted","Data":"d7ea456459ea6c8776e534771189487e002208e4855ef99ac74af6758f992452"} Oct 31 22:15:02.296709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.296684 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerStarted","Data":"90da6f35e773317e16b6f8c9a471550b38ea92083474183958d42626eb6200cd"} Oct 31 22:15:02.298576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.298522 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" event={"ID":"3ee08211-d158-43a4-a055-61d92d2ee655","Type":"ContainerStarted","Data":"84710a3538c750d0ff164e4d8206799f067defc901a9b6cb4ca4994722167bc3"} Oct 31 22:15:02.300363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.300332 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" event={"ID":"7c519470-3a10-488d-a542-20423d52211a","Type":"ContainerStarted","Data":"4117c71e9e62efabfe1be1e226d77f0ed2f6691091d164eda316eeb8bbd576cd"} Oct 31 22:15:02.300626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.300387 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" event={"ID":"7c519470-3a10-488d-a542-20423d52211a","Type":"ContainerStarted","Data":"87aa8775bac7a50f5c663a731d7e9e1290984127a0591c1910a10336e869469c"} Oct 31 22:15:02.302351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.302330 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerStarted","Data":"e00a5adca90a0b780b5fa2858c707417dd11bcbed3fb535b24db524fa72590cd"} Oct 31 22:15:02.302453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.302383 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerStarted","Data":"d54c0ab4b7d599e152054f488c80536ddf251c3171aaebc9ec799d00aa288bea"} Oct 31 22:15:02.303987 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.303965 4294 generic.go:356] "Generic (PLEG): container finished" podID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerID="b529033d33490927ad259c59fedd96f8d4b0cb213cb1cc376dfdd457d1ce9852" exitCode=60 Oct 31 22:15:02.304093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.304014 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" event={"ID":"a541b3ad-0cf9-4a2e-b488-f2d54d250c83","Type":"ContainerDied","Data":"b529033d33490927ad259c59fedd96f8d4b0cb213cb1cc376dfdd457d1ce9852"} Oct 31 22:15:02.304410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.304388 4294 scope.go:122] "RemoveContainer" containerID="b529033d33490927ad259c59fedd96f8d4b0cb213cb1cc376dfdd457d1ce9852" Oct 31 22:15:02.305862 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.305843 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerStarted","Data":"41dacafcfbe056660282241deaa17ac123544acaa314305e5dbe95cd16985762"} Oct 31 22:15:02.305913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.305888 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerStarted","Data":"e7d8d8e2269f1a4e0c04db8a676084764c3ed6b739d8e1e13f275b9691a43f2f"} Oct 31 22:15:02.307459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.307433 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" event={"ID":"728fa16a-ae89-4220-8990-7ac122157f07","Type":"ContainerStarted","Data":"5273aa262c0a4e5c13b26c802a8baf546e0c4b162bdd79f8ce4fd89f58562ebb"} Oct 31 22:15:02.308959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.308937 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" event={"ID":"9f0741c6-c011-4c39-b014-0d9034978b03","Type":"ContainerStarted","Data":"3341cd0f7bbe47fd2db59f0d76fafc915e481172bdeeac8ac9a7efb40b3948bb"} Oct 31 22:15:02.310562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.310541 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" event={"ID":"f6e5c17f-0945-4321-b1d4-26c06bfd778c","Type":"ContainerStarted","Data":"1112abcbaf3218176352292c5b0c33894d0b75753e3a04b7ccdb710114e69ba3"} Oct 31 22:15:02.312175 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.312155 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" event={"ID":"31db38fe-c0de-4701-b7ed-62b8a7539a1a","Type":"ContainerStarted","Data":"cc08b4d7149ff4676269e8dcf01578f124f15a08b64b571a39eff598488674ab"} Oct 31 22:15:02.314053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.314027 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" event={"ID":"3aa3eb2b-4e44-4796-904f-a9aec7047555","Type":"ContainerStarted","Data":"8b3fe72a6ed98d74b3e0925ba9597dc6d9d5fdc0d6c24ab6024cf928ac6111a9"} Oct 31 22:15:02.315994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.315972 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" event={"ID":"8db9dfad-1040-4e7c-8203-1497b21954f0","Type":"ContainerStarted","Data":"6530ed4dbd2b044fa976ded27fe2202fdd16f8728255092c7ee6674319a0c2c7"} Oct 31 22:15:02.317719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.317698 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerStarted","Data":"0b59332bf7569dde29f9234d7ed70f493a16f23830d6ebe29859713856ded243"} Oct 31 22:15:02.317824 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.317734 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerStarted","Data":"b0a8d5e81705252b36fdaba40393c29a63375448b5da614967ada26e505cb9bb"} Oct 31 22:15:02.319399 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.319378 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerStarted","Data":"0cd416ca63dcf6c7d439d156b14b8b913dd8bdbc20475eea59de5d0e7ea4df95"} Oct 31 22:15:02.319491 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.319418 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerStarted","Data":"537efebfe3a05725212fc2f09fe1c9d6dfd3af088ed3f5288413c60564750a5c"} Oct 31 22:15:02.341603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.341563 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" podStartSLOduration=2.341552106 podStartE2EDuration="2.341552106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.339166838 +0000 UTC m=+3542.464274069" watchObservedRunningTime="2026-10-31 22:15:02.341552106 +0000 UTC m=+3542.466659337" Oct 31 22:15:02.465527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.465427 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" podStartSLOduration=2.465413506 podStartE2EDuration="2.465413506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.424252323 +0000 UTC m=+3542.549359553" watchObservedRunningTime="2026-10-31 22:15:02.465413506 +0000 UTC m=+3542.590520736" Oct 31 22:15:02.465688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.465634 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" podStartSLOduration=2.465628706 podStartE2EDuration="2.465628706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.463649405 +0000 UTC m=+3542.588756637" watchObservedRunningTime="2026-10-31 22:15:02.465628706 +0000 UTC m=+3542.590735972" Oct 31 22:15:02.505989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.505937 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" podStartSLOduration=2.505923927 podStartE2EDuration="2.505923927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.502827747 +0000 UTC m=+3542.627934981" watchObservedRunningTime="2026-10-31 22:15:02.505923927 +0000 UTC m=+3542.631031156" Oct 31 22:15:02.545343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.545257 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" podStartSLOduration=2.545241169 podStartE2EDuration="2.545241169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.541916755 +0000 UTC m=+3542.667023991" watchObservedRunningTime="2026-10-31 22:15:02.545241169 +0000 UTC m=+3542.670348399" Oct 31 22:15:02.585533 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.585488 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" podStartSLOduration=2.5854758479999997 podStartE2EDuration="2.585475848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.580808199 +0000 UTC m=+3542.705915434" watchObservedRunningTime="2026-10-31 22:15:02.585475848 +0000 UTC m=+3542.710583078" Oct 31 22:15:02.619962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.619919 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" podStartSLOduration=2.619898107 podStartE2EDuration="2.619898107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.619565911 +0000 UTC m=+3542.744673143" watchObservedRunningTime="2026-10-31 22:15:02.619898107 +0000 UTC m=+3542.745005339" Oct 31 22:15:02.664471 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.664426 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" podStartSLOduration=2.664413933 podStartE2EDuration="2.664413933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.66138564 +0000 UTC m=+3542.786492870" watchObservedRunningTime="2026-10-31 22:15:02.664413933 +0000 UTC m=+3542.789521163" Oct 31 22:15:02.702852 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.702808 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" podStartSLOduration=2.7027950880000002 podStartE2EDuration="2.702795088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.702211714 +0000 UTC m=+3542.827318955" watchObservedRunningTime="2026-10-31 22:15:02.702795088 +0000 UTC m=+3542.827902317" Oct 31 22:15:02.743147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.743093 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" podStartSLOduration=2.743080531 podStartE2EDuration="2.743080531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.742370393 +0000 UTC m=+3542.867477638" watchObservedRunningTime="2026-10-31 22:15:02.743080531 +0000 UTC m=+3542.868187794" Oct 31 22:15:02.788175 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.788120 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" podStartSLOduration=2.788102121 podStartE2EDuration="2.788102121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.783826689 +0000 UTC m=+3542.908933924" watchObservedRunningTime="2026-10-31 22:15:02.788102121 +0000 UTC m=+3542.913209352" Oct 31 22:15:02.828442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.828397 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" podStartSLOduration=2.828378656 podStartE2EDuration="2.828378656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.826154972 +0000 UTC m=+3542.951262215" watchObservedRunningTime="2026-10-31 22:15:02.828378656 +0000 UTC m=+3542.953485887" Oct 31 22:15:02.868636 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:02.868594 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" podStartSLOduration=2.868580799 podStartE2EDuration="2.868580799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:02.868416834 +0000 UTC m=+3542.993524064" watchObservedRunningTime="2026-10-31 22:15:02.868580799 +0000 UTC m=+3542.993688029" Oct 31 22:15:03.330931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.330904 4294 generic.go:356] "Generic (PLEG): container finished" podID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerID="0b59332bf7569dde29f9234d7ed70f493a16f23830d6ebe29859713856ded243" exitCode=60 Oct 31 22:15:03.331347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.330986 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerDied","Data":"0b59332bf7569dde29f9234d7ed70f493a16f23830d6ebe29859713856ded243"} Oct 31 22:15:03.331347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.331266 4294 scope.go:122] "RemoveContainer" containerID="0b59332bf7569dde29f9234d7ed70f493a16f23830d6ebe29859713856ded243" Oct 31 22:15:03.332867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.332760 4294 generic.go:356] "Generic (PLEG): container finished" podID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerID="0cd416ca63dcf6c7d439d156b14b8b913dd8bdbc20475eea59de5d0e7ea4df95" exitCode=60 Oct 31 22:15:03.332867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.332848 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerDied","Data":"0cd416ca63dcf6c7d439d156b14b8b913dd8bdbc20475eea59de5d0e7ea4df95"} Oct 31 22:15:03.333075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.333027 4294 scope.go:122] "RemoveContainer" containerID="0cd416ca63dcf6c7d439d156b14b8b913dd8bdbc20475eea59de5d0e7ea4df95" Oct 31 22:15:03.334706 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.334678 4294 generic.go:356] "Generic (PLEG): container finished" podID="4bc62938-ef10-4235-91af-d518a045ce95" containerID="d7ea456459ea6c8776e534771189487e002208e4855ef99ac74af6758f992452" exitCode=60 Oct 31 22:15:03.334803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.334746 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerDied","Data":"d7ea456459ea6c8776e534771189487e002208e4855ef99ac74af6758f992452"} Oct 31 22:15:03.334984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.334962 4294 scope.go:122] "RemoveContainer" containerID="d7ea456459ea6c8776e534771189487e002208e4855ef99ac74af6758f992452" Oct 31 22:15:03.336750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.336710 4294 generic.go:356] "Generic (PLEG): container finished" podID="3ee08211-d158-43a4-a055-61d92d2ee655" containerID="84710a3538c750d0ff164e4d8206799f067defc901a9b6cb4ca4994722167bc3" exitCode=60 Oct 31 22:15:03.336804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.336788 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" event={"ID":"3ee08211-d158-43a4-a055-61d92d2ee655","Type":"ContainerDied","Data":"84710a3538c750d0ff164e4d8206799f067defc901a9b6cb4ca4994722167bc3"} Oct 31 22:15:03.336986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.336971 4294 scope.go:122] "RemoveContainer" containerID="84710a3538c750d0ff164e4d8206799f067defc901a9b6cb4ca4994722167bc3" Oct 31 22:15:03.338479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.338460 4294 generic.go:356] "Generic (PLEG): container finished" podID="7c519470-3a10-488d-a542-20423d52211a" containerID="4117c71e9e62efabfe1be1e226d77f0ed2f6691091d164eda316eeb8bbd576cd" exitCode=60 Oct 31 22:15:03.338577 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.338500 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" event={"ID":"7c519470-3a10-488d-a542-20423d52211a","Type":"ContainerDied","Data":"4117c71e9e62efabfe1be1e226d77f0ed2f6691091d164eda316eeb8bbd576cd"} Oct 31 22:15:03.338730 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.338714 4294 scope.go:122] "RemoveContainer" containerID="4117c71e9e62efabfe1be1e226d77f0ed2f6691091d164eda316eeb8bbd576cd" Oct 31 22:15:03.340083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.340063 4294 generic.go:356] "Generic (PLEG): container finished" podID="4312b8c4-a53c-4354-b696-695e71f214d6" containerID="e00a5adca90a0b780b5fa2858c707417dd11bcbed3fb535b24db524fa72590cd" exitCode=60 Oct 31 22:15:03.340158 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.340127 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerDied","Data":"e00a5adca90a0b780b5fa2858c707417dd11bcbed3fb535b24db524fa72590cd"} Oct 31 22:15:03.340284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.340271 4294 scope.go:122] "RemoveContainer" containerID="e00a5adca90a0b780b5fa2858c707417dd11bcbed3fb535b24db524fa72590cd" Oct 31 22:15:03.341957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.341935 4294 generic.go:356] "Generic (PLEG): container finished" podID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerID="428749d24b091dd7d13175715d522f991a7beaf9053d59b5e9af81612d4af8b2" exitCode=60 Oct 31 22:15:03.342031 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.342008 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" event={"ID":"a541b3ad-0cf9-4a2e-b488-f2d54d250c83","Type":"ContainerDied","Data":"428749d24b091dd7d13175715d522f991a7beaf9053d59b5e9af81612d4af8b2"} Oct 31 22:15:03.342107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.342055 4294 scope.go:122] "RemoveContainer" containerID="b529033d33490927ad259c59fedd96f8d4b0cb213cb1cc376dfdd457d1ce9852" Oct 31 22:15:03.342217 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.342198 4294 scope.go:122] "RemoveContainer" containerID="428749d24b091dd7d13175715d522f991a7beaf9053d59b5e9af81612d4af8b2" Oct 31 22:15:03.342471 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:03.342450 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc_odh-ai-gateway-infra(a541b3ad-0cf9-4a2e-b488-f2d54d250c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" Oct 31 22:15:03.345189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.344971 4294 generic.go:356] "Generic (PLEG): container finished" podID="845f9969-a52b-4d80-a641-81c2e8020c54" containerID="41dacafcfbe056660282241deaa17ac123544acaa314305e5dbe95cd16985762" exitCode=60 Oct 31 22:15:03.345189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.344996 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerDied","Data":"41dacafcfbe056660282241deaa17ac123544acaa314305e5dbe95cd16985762"} Oct 31 22:15:03.345428 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.345235 4294 scope.go:122] "RemoveContainer" containerID="41dacafcfbe056660282241deaa17ac123544acaa314305e5dbe95cd16985762" Oct 31 22:15:03.346668 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.346642 4294 generic.go:356] "Generic (PLEG): container finished" podID="728fa16a-ae89-4220-8990-7ac122157f07" containerID="ca547d275c820a35ff787398110c3c8f1e4db7a0ae9a2f456937c127ea776369" exitCode=60 Oct 31 22:15:03.346759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.346749 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" event={"ID":"728fa16a-ae89-4220-8990-7ac122157f07","Type":"ContainerDied","Data":"ca547d275c820a35ff787398110c3c8f1e4db7a0ae9a2f456937c127ea776369"} Oct 31 22:15:03.346975 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.346962 4294 scope.go:122] "RemoveContainer" containerID="ca547d275c820a35ff787398110c3c8f1e4db7a0ae9a2f456937c127ea776369" Oct 31 22:15:03.348761 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.348739 4294 generic.go:356] "Generic (PLEG): container finished" podID="9f0741c6-c011-4c39-b014-0d9034978b03" containerID="f6d8cef9d94060d363135906dab4b6e713ef26893a08634b863575e22327c7cb" exitCode=60 Oct 31 22:15:03.348967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.348947 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" event={"ID":"9f0741c6-c011-4c39-b014-0d9034978b03","Type":"ContainerDied","Data":"f6d8cef9d94060d363135906dab4b6e713ef26893a08634b863575e22327c7cb"} Oct 31 22:15:03.349092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.349074 4294 scope.go:122] "RemoveContainer" containerID="f6d8cef9d94060d363135906dab4b6e713ef26893a08634b863575e22327c7cb" Oct 31 22:15:03.350511 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.350489 4294 generic.go:356] "Generic (PLEG): container finished" podID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerID="1112abcbaf3218176352292c5b0c33894d0b75753e3a04b7ccdb710114e69ba3" exitCode=60 Oct 31 22:15:03.350657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.350570 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" event={"ID":"f6e5c17f-0945-4321-b1d4-26c06bfd778c","Type":"ContainerDied","Data":"1112abcbaf3218176352292c5b0c33894d0b75753e3a04b7ccdb710114e69ba3"} Oct 31 22:15:03.350751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.350732 4294 scope.go:122] "RemoveContainer" containerID="1112abcbaf3218176352292c5b0c33894d0b75753e3a04b7ccdb710114e69ba3" Oct 31 22:15:03.352240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.352224 4294 generic.go:356] "Generic (PLEG): container finished" podID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerID="6177aef64372b80115286fa248945e71bad11cfd6e49a83283a424780a702989" exitCode=60 Oct 31 22:15:03.352311 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.352284 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" event={"ID":"31db38fe-c0de-4701-b7ed-62b8a7539a1a","Type":"ContainerDied","Data":"6177aef64372b80115286fa248945e71bad11cfd6e49a83283a424780a702989"} Oct 31 22:15:03.352590 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.352531 4294 scope.go:122] "RemoveContainer" containerID="6177aef64372b80115286fa248945e71bad11cfd6e49a83283a424780a702989" Oct 31 22:15:03.355994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.355877 4294 generic.go:356] "Generic (PLEG): container finished" podID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerID="65195df433a2bbf8d1cdbfd0e0202123aa96139e878ebaab0c1c50a414239a23" exitCode=60 Oct 31 22:15:03.355994 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.355958 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" event={"ID":"3aa3eb2b-4e44-4796-904f-a9aec7047555","Type":"ContainerDied","Data":"65195df433a2bbf8d1cdbfd0e0202123aa96139e878ebaab0c1c50a414239a23"} Oct 31 22:15:03.356181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.356144 4294 scope.go:122] "RemoveContainer" containerID="65195df433a2bbf8d1cdbfd0e0202123aa96139e878ebaab0c1c50a414239a23" Oct 31 22:15:03.357826 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.357807 4294 generic.go:356] "Generic (PLEG): container finished" podID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerID="90d62a114d55dd996abcbe93f0ffa365a859d91c0cb18fb46b7154e4d06dc7c2" exitCode=60 Oct 31 22:15:03.357900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.357838 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" event={"ID":"8db9dfad-1040-4e7c-8203-1497b21954f0","Type":"ContainerDied","Data":"90d62a114d55dd996abcbe93f0ffa365a859d91c0cb18fb46b7154e4d06dc7c2"} Oct 31 22:15:03.358042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.358029 4294 scope.go:122] "RemoveContainer" containerID="90d62a114d55dd996abcbe93f0ffa365a859d91c0cb18fb46b7154e4d06dc7c2" Oct 31 22:15:03.395521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.395495 4294 scope.go:122] "RemoveContainer" containerID="5f7d1db76ebb2c997ecd5aedb8f7e4c8d70a9edea17115d371e034c846dabe3a" Oct 31 22:15:03.480151 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.480027 4294 scope.go:122] "RemoveContainer" containerID="1f3753a46fabe8545a30a8a90f319a598de03f8ce63d281c07f8ca082de7ba29" Oct 31 22:15:03.649208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.649185 4294 scope.go:122] "RemoveContainer" containerID="f3badaf30d2a01749150ae2ac0da30f59b0a9e8153bbeace492dcca59d247e36" Oct 31 22:15:03.661515 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.661491 4294 scope.go:122] "RemoveContainer" containerID="3e4aaa112f22396c10928cf47dbfd5318365074e04ca125d16f27c801ae5070c" Oct 31 22:15:03.673184 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:03.673157 4294 scope.go:122] "RemoveContainer" containerID="07981f1b75b8c9bb4e838804384ac1a21ef3d6292f3cc24538d377a22c7c6d46" Oct 31 22:15:04.368947 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.368916 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" event={"ID":"9f0741c6-c011-4c39-b014-0d9034978b03","Type":"ContainerStarted","Data":"40e750eef599b5fd4d075a7fedc20fe3511dc71ee5ad49a09fb4b994e52f8f41"} Oct 31 22:15:04.371084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.371057 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" event={"ID":"f6e5c17f-0945-4321-b1d4-26c06bfd778c","Type":"ContainerStarted","Data":"bcc322a72b8d5c81f85a9676212dd8915bf390ab72ff2dfc60647d34ee8c6bb9"} Oct 31 22:15:04.378397 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.378222 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" event={"ID":"31db38fe-c0de-4701-b7ed-62b8a7539a1a","Type":"ContainerStarted","Data":"98823e9c6baa0d1295de03bed96254125a1a67a6481e2b435717faef81145e1a"} Oct 31 22:15:04.380688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.380657 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" event={"ID":"3aa3eb2b-4e44-4796-904f-a9aec7047555","Type":"ContainerStarted","Data":"48ff12bdf699909d80879b04e9b89a8a101775e05c2351781304ecd192df4a40"} Oct 31 22:15:04.383441 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.383415 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" event={"ID":"8db9dfad-1040-4e7c-8203-1497b21954f0","Type":"ContainerStarted","Data":"48ff77df7eb4efae05fee31bef733405dd166ba3ae4e7711aaf581f170a21719"} Oct 31 22:15:04.385453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.385434 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerStarted","Data":"01d602db5500c43ec5ef3352720bd1cd5bc29c14ea3ec671397497df9d081b65"} Oct 31 22:15:04.387254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.387234 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerStarted","Data":"be8083c7f06b3a3c0074649da93e0ca57954a2847f493357c6d10107bd50a036"} Oct 31 22:15:04.388923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.388901 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerStarted","Data":"ee98f272dd6963c175343b5fbc1e073ce17cae58ef12aca607a5b231fd7c2e5f"} Oct 31 22:15:04.390716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.390698 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" event={"ID":"3ee08211-d158-43a4-a055-61d92d2ee655","Type":"ContainerStarted","Data":"024f3f84fe050b52a5264ca7582a9338173e22c605c461d3167a26582c2ffc33"} Oct 31 22:15:04.392473 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.392454 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" event={"ID":"7c519470-3a10-488d-a542-20423d52211a","Type":"ContainerStarted","Data":"d680dced1ea449bfd7248b30c5a42f6d47eabc6c5a0e37af6b72f24d2b4fb281"} Oct 31 22:15:04.394468 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.394448 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerStarted","Data":"815fc56b19ac477b19e7e7d68094e320d25df0e1b680d706761786cd844ab15f"} Oct 31 22:15:04.396166 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.396149 4294 scope.go:122] "RemoveContainer" containerID="428749d24b091dd7d13175715d522f991a7beaf9053d59b5e9af81612d4af8b2" Oct 31 22:15:04.396420 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:04.396399 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc_odh-ai-gateway-infra(a541b3ad-0cf9-4a2e-b488-f2d54d250c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" Oct 31 22:15:04.397697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.397671 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerStarted","Data":"6c8f7fd456e375208655ccb08a2f07d2a4d8d3c5fd72a770401fa3f62d95a869"} Oct 31 22:15:04.399451 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:04.399431 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" event={"ID":"728fa16a-ae89-4220-8990-7ac122157f07","Type":"ContainerStarted","Data":"a9e4c772d74698ed9488bafed7708032d26091f93531efa0974115a057b449b3"} Oct 31 22:15:05.409654 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.409556 4294 generic.go:356] "Generic (PLEG): container finished" podID="4312b8c4-a53c-4354-b696-695e71f214d6" containerID="815fc56b19ac477b19e7e7d68094e320d25df0e1b680d706761786cd844ab15f" exitCode=60 Oct 31 22:15:05.410279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.409640 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerDied","Data":"815fc56b19ac477b19e7e7d68094e320d25df0e1b680d706761786cd844ab15f"} Oct 31 22:15:05.410279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.409750 4294 scope.go:122] "RemoveContainer" containerID="e00a5adca90a0b780b5fa2858c707417dd11bcbed3fb535b24db524fa72590cd" Oct 31 22:15:05.410279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.409948 4294 scope.go:122] "RemoveContainer" containerID="815fc56b19ac477b19e7e7d68094e320d25df0e1b680d706761786cd844ab15f" Oct 31 22:15:05.410454 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.410411 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt_odh-ai-gateway-infra(4312b8c4-a53c-4354-b696-695e71f214d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" Oct 31 22:15:05.411939 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.411920 4294 generic.go:356] "Generic (PLEG): container finished" podID="845f9969-a52b-4d80-a641-81c2e8020c54" containerID="6c8f7fd456e375208655ccb08a2f07d2a4d8d3c5fd72a770401fa3f62d95a869" exitCode=60 Oct 31 22:15:05.412018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.411998 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerDied","Data":"6c8f7fd456e375208655ccb08a2f07d2a4d8d3c5fd72a770401fa3f62d95a869"} Oct 31 22:15:05.412226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.412211 4294 scope.go:122] "RemoveContainer" containerID="6c8f7fd456e375208655ccb08a2f07d2a4d8d3c5fd72a770401fa3f62d95a869" Oct 31 22:15:05.412576 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.412556 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4_odh-ai-gateway-infra(845f9969-a52b-4d80-a641-81c2e8020c54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" Oct 31 22:15:05.413792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.413773 4294 generic.go:356] "Generic (PLEG): container finished" podID="728fa16a-ae89-4220-8990-7ac122157f07" containerID="a9e4c772d74698ed9488bafed7708032d26091f93531efa0974115a057b449b3" exitCode=60 Oct 31 22:15:05.413866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.413840 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" event={"ID":"728fa16a-ae89-4220-8990-7ac122157f07","Type":"ContainerDied","Data":"a9e4c772d74698ed9488bafed7708032d26091f93531efa0974115a057b449b3"} Oct 31 22:15:05.414119 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.414101 4294 scope.go:122] "RemoveContainer" containerID="a9e4c772d74698ed9488bafed7708032d26091f93531efa0974115a057b449b3" Oct 31 22:15:05.414503 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.414480 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn_odh-ai-gateway-infra(728fa16a-ae89-4220-8990-7ac122157f07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" podUID="728fa16a-ae89-4220-8990-7ac122157f07" Oct 31 22:15:05.415804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.415784 4294 generic.go:356] "Generic (PLEG): container finished" podID="9f0741c6-c011-4c39-b014-0d9034978b03" containerID="40e750eef599b5fd4d075a7fedc20fe3511dc71ee5ad49a09fb4b994e52f8f41" exitCode=60 Oct 31 22:15:05.415883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.415864 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" event={"ID":"9f0741c6-c011-4c39-b014-0d9034978b03","Type":"ContainerDied","Data":"40e750eef599b5fd4d075a7fedc20fe3511dc71ee5ad49a09fb4b994e52f8f41"} Oct 31 22:15:05.416054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.416039 4294 scope.go:122] "RemoveContainer" containerID="40e750eef599b5fd4d075a7fedc20fe3511dc71ee5ad49a09fb4b994e52f8f41" Oct 31 22:15:05.416365 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.416341 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw_odh-ai-gateway-infra(9f0741c6-c011-4c39-b014-0d9034978b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" Oct 31 22:15:05.417645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.417623 4294 generic.go:356] "Generic (PLEG): container finished" podID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerID="bcc322a72b8d5c81f85a9676212dd8915bf390ab72ff2dfc60647d34ee8c6bb9" exitCode=60 Oct 31 22:15:05.417713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.417668 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" event={"ID":"f6e5c17f-0945-4321-b1d4-26c06bfd778c","Type":"ContainerDied","Data":"bcc322a72b8d5c81f85a9676212dd8915bf390ab72ff2dfc60647d34ee8c6bb9"} Oct 31 22:15:05.417900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.417880 4294 scope.go:122] "RemoveContainer" containerID="bcc322a72b8d5c81f85a9676212dd8915bf390ab72ff2dfc60647d34ee8c6bb9" Oct 31 22:15:05.418191 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.418165 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp_odh-ai-gateway-infra(f6e5c17f-0945-4321-b1d4-26c06bfd778c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" Oct 31 22:15:05.419617 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.419598 4294 generic.go:356] "Generic (PLEG): container finished" podID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerID="98823e9c6baa0d1295de03bed96254125a1a67a6481e2b435717faef81145e1a" exitCode=60 Oct 31 22:15:05.419712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.419661 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" event={"ID":"31db38fe-c0de-4701-b7ed-62b8a7539a1a","Type":"ContainerDied","Data":"98823e9c6baa0d1295de03bed96254125a1a67a6481e2b435717faef81145e1a"} Oct 31 22:15:05.419893 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.419880 4294 scope.go:122] "RemoveContainer" containerID="98823e9c6baa0d1295de03bed96254125a1a67a6481e2b435717faef81145e1a" Oct 31 22:15:05.420116 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.420094 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz_odh-ai-gateway-infra(31db38fe-c0de-4701-b7ed-62b8a7539a1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" Oct 31 22:15:05.421671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.421653 4294 generic.go:356] "Generic (PLEG): container finished" podID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerID="48ff12bdf699909d80879b04e9b89a8a101775e05c2351781304ecd192df4a40" exitCode=60 Oct 31 22:15:05.421763 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.421681 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" event={"ID":"3aa3eb2b-4e44-4796-904f-a9aec7047555","Type":"ContainerDied","Data":"48ff12bdf699909d80879b04e9b89a8a101775e05c2351781304ecd192df4a40"} Oct 31 22:15:05.421946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.421928 4294 scope.go:122] "RemoveContainer" containerID="48ff12bdf699909d80879b04e9b89a8a101775e05c2351781304ecd192df4a40" Oct 31 22:15:05.422219 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.422197 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv_odh-ai-gateway-infra(3aa3eb2b-4e44-4796-904f-a9aec7047555)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" Oct 31 22:15:05.423552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.423534 4294 generic.go:356] "Generic (PLEG): container finished" podID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerID="48ff77df7eb4efae05fee31bef733405dd166ba3ae4e7711aaf581f170a21719" exitCode=60 Oct 31 22:15:05.423642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.423602 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" event={"ID":"8db9dfad-1040-4e7c-8203-1497b21954f0","Type":"ContainerDied","Data":"48ff77df7eb4efae05fee31bef733405dd166ba3ae4e7711aaf581f170a21719"} Oct 31 22:15:05.423846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.423826 4294 scope.go:122] "RemoveContainer" containerID="48ff77df7eb4efae05fee31bef733405dd166ba3ae4e7711aaf581f170a21719" Oct 31 22:15:05.424147 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.424119 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn_odh-ai-gateway-infra(8db9dfad-1040-4e7c-8203-1497b21954f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" Oct 31 22:15:05.424993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.424973 4294 scope.go:122] "RemoveContainer" containerID="41dacafcfbe056660282241deaa17ac123544acaa314305e5dbe95cd16985762" Oct 31 22:15:05.425555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.425534 4294 generic.go:356] "Generic (PLEG): container finished" podID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerID="01d602db5500c43ec5ef3352720bd1cd5bc29c14ea3ec671397497df9d081b65" exitCode=60 Oct 31 22:15:05.425652 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.425558 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerDied","Data":"01d602db5500c43ec5ef3352720bd1cd5bc29c14ea3ec671397497df9d081b65"} Oct 31 22:15:05.425812 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.425794 4294 scope.go:122] "RemoveContainer" containerID="01d602db5500c43ec5ef3352720bd1cd5bc29c14ea3ec671397497df9d081b65" Oct 31 22:15:05.426272 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.426249 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd_odh-ai-gateway-infra(d7a1dfb1-a6f3-42fa-8566-bb614078045a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" Oct 31 22:15:05.427519 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.427499 4294 generic.go:356] "Generic (PLEG): container finished" podID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerID="be8083c7f06b3a3c0074649da93e0ca57954a2847f493357c6d10107bd50a036" exitCode=60 Oct 31 22:15:05.427605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.427574 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerDied","Data":"be8083c7f06b3a3c0074649da93e0ca57954a2847f493357c6d10107bd50a036"} Oct 31 22:15:05.427781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.427762 4294 scope.go:122] "RemoveContainer" containerID="be8083c7f06b3a3c0074649da93e0ca57954a2847f493357c6d10107bd50a036" Oct 31 22:15:05.427998 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.427964 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g_odh-ai-gateway-infra(d5a772d8-9d68-4a2f-a241-6f356eb5a2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" Oct 31 22:15:05.429519 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.429495 4294 generic.go:356] "Generic (PLEG): container finished" podID="4bc62938-ef10-4235-91af-d518a045ce95" containerID="ee98f272dd6963c175343b5fbc1e073ce17cae58ef12aca607a5b231fd7c2e5f" exitCode=60 Oct 31 22:15:05.429612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.429570 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerDied","Data":"ee98f272dd6963c175343b5fbc1e073ce17cae58ef12aca607a5b231fd7c2e5f"} Oct 31 22:15:05.429754 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.429735 4294 scope.go:122] "RemoveContainer" containerID="ee98f272dd6963c175343b5fbc1e073ce17cae58ef12aca607a5b231fd7c2e5f" Oct 31 22:15:05.429987 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.429969 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm_odh-ai-gateway-infra(4bc62938-ef10-4235-91af-d518a045ce95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" podUID="4bc62938-ef10-4235-91af-d518a045ce95" Oct 31 22:15:05.431589 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.431565 4294 generic.go:356] "Generic (PLEG): container finished" podID="3ee08211-d158-43a4-a055-61d92d2ee655" containerID="024f3f84fe050b52a5264ca7582a9338173e22c605c461d3167a26582c2ffc33" exitCode=60 Oct 31 22:15:05.431661 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.431597 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" event={"ID":"3ee08211-d158-43a4-a055-61d92d2ee655","Type":"ContainerDied","Data":"024f3f84fe050b52a5264ca7582a9338173e22c605c461d3167a26582c2ffc33"} Oct 31 22:15:05.431866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.431847 4294 scope.go:122] "RemoveContainer" containerID="024f3f84fe050b52a5264ca7582a9338173e22c605c461d3167a26582c2ffc33" Oct 31 22:15:05.432161 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.432138 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms_odh-ai-gateway-infra(3ee08211-d158-43a4-a055-61d92d2ee655)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" Oct 31 22:15:05.433510 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.433488 4294 generic.go:356] "Generic (PLEG): container finished" podID="7c519470-3a10-488d-a542-20423d52211a" containerID="d680dced1ea449bfd7248b30c5a42f6d47eabc6c5a0e37af6b72f24d2b4fb281" exitCode=60 Oct 31 22:15:05.433595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.433524 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" event={"ID":"7c519470-3a10-488d-a542-20423d52211a","Type":"ContainerDied","Data":"d680dced1ea449bfd7248b30c5a42f6d47eabc6c5a0e37af6b72f24d2b4fb281"} Oct 31 22:15:05.433740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.433726 4294 scope.go:122] "RemoveContainer" containerID="d680dced1ea449bfd7248b30c5a42f6d47eabc6c5a0e37af6b72f24d2b4fb281" Oct 31 22:15:05.433918 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:05.433903 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt_odh-ai-gateway-infra(7c519470-3a10-488d-a542-20423d52211a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" podUID="7c519470-3a10-488d-a542-20423d52211a" Oct 31 22:15:05.438999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.438982 4294 scope.go:122] "RemoveContainer" containerID="ca547d275c820a35ff787398110c3c8f1e4db7a0ae9a2f456937c127ea776369" Oct 31 22:15:05.453215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.453195 4294 scope.go:122] "RemoveContainer" containerID="f6d8cef9d94060d363135906dab4b6e713ef26893a08634b863575e22327c7cb" Oct 31 22:15:05.467057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.467036 4294 scope.go:122] "RemoveContainer" containerID="1112abcbaf3218176352292c5b0c33894d0b75753e3a04b7ccdb710114e69ba3" Oct 31 22:15:05.480177 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.480158 4294 scope.go:122] "RemoveContainer" containerID="6177aef64372b80115286fa248945e71bad11cfd6e49a83283a424780a702989" Oct 31 22:15:05.505254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.505227 4294 scope.go:122] "RemoveContainer" containerID="65195df433a2bbf8d1cdbfd0e0202123aa96139e878ebaab0c1c50a414239a23" Oct 31 22:15:05.518790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.518769 4294 scope.go:122] "RemoveContainer" containerID="90d62a114d55dd996abcbe93f0ffa365a859d91c0cb18fb46b7154e4d06dc7c2" Oct 31 22:15:05.532007 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.531984 4294 scope.go:122] "RemoveContainer" containerID="0b59332bf7569dde29f9234d7ed70f493a16f23830d6ebe29859713856ded243" Oct 31 22:15:05.546125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.546104 4294 scope.go:122] "RemoveContainer" containerID="0cd416ca63dcf6c7d439d156b14b8b913dd8bdbc20475eea59de5d0e7ea4df95" Oct 31 22:15:05.558914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.558897 4294 scope.go:122] "RemoveContainer" containerID="d7ea456459ea6c8776e534771189487e002208e4855ef99ac74af6758f992452" Oct 31 22:15:05.572386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.572364 4294 scope.go:122] "RemoveContainer" containerID="84710a3538c750d0ff164e4d8206799f067defc901a9b6cb4ca4994722167bc3" Oct 31 22:15:05.585545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:05.585526 4294 scope.go:122] "RemoveContainer" containerID="4117c71e9e62efabfe1be1e226d77f0ed2f6691091d164eda316eeb8bbd576cd" Oct 31 22:15:06.443018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.442988 4294 scope.go:122] "RemoveContainer" containerID="40e750eef599b5fd4d075a7fedc20fe3511dc71ee5ad49a09fb4b994e52f8f41" Oct 31 22:15:06.443480 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.443411 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw_odh-ai-gateway-infra(9f0741c6-c011-4c39-b014-0d9034978b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" Oct 31 22:15:06.444715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.444691 4294 scope.go:122] "RemoveContainer" containerID="bcc322a72b8d5c81f85a9676212dd8915bf390ab72ff2dfc60647d34ee8c6bb9" Oct 31 22:15:06.445039 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.445017 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp_odh-ai-gateway-infra(f6e5c17f-0945-4321-b1d4-26c06bfd778c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" Oct 31 22:15:06.446400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.446379 4294 scope.go:122] "RemoveContainer" containerID="98823e9c6baa0d1295de03bed96254125a1a67a6481e2b435717faef81145e1a" Oct 31 22:15:06.446705 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.446676 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz_odh-ai-gateway-infra(31db38fe-c0de-4701-b7ed-62b8a7539a1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" Oct 31 22:15:06.448011 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.447993 4294 scope.go:122] "RemoveContainer" containerID="48ff12bdf699909d80879b04e9b89a8a101775e05c2351781304ecd192df4a40" Oct 31 22:15:06.448262 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.448231 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv_odh-ai-gateway-infra(3aa3eb2b-4e44-4796-904f-a9aec7047555)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" Oct 31 22:15:06.449621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.449603 4294 scope.go:122] "RemoveContainer" containerID="48ff77df7eb4efae05fee31bef733405dd166ba3ae4e7711aaf581f170a21719" Oct 31 22:15:06.449831 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.449806 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn_odh-ai-gateway-infra(8db9dfad-1040-4e7c-8203-1497b21954f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" Oct 31 22:15:06.451097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.451080 4294 scope.go:122] "RemoveContainer" containerID="01d602db5500c43ec5ef3352720bd1cd5bc29c14ea3ec671397497df9d081b65" Oct 31 22:15:06.451319 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.451286 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd_odh-ai-gateway-infra(d7a1dfb1-a6f3-42fa-8566-bb614078045a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" Oct 31 22:15:06.452557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.452539 4294 scope.go:122] "RemoveContainer" containerID="be8083c7f06b3a3c0074649da93e0ca57954a2847f493357c6d10107bd50a036" Oct 31 22:15:06.452739 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.452718 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g_odh-ai-gateway-infra(d5a772d8-9d68-4a2f-a241-6f356eb5a2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" Oct 31 22:15:06.454064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.454044 4294 scope.go:122] "RemoveContainer" containerID="ee98f272dd6963c175343b5fbc1e073ce17cae58ef12aca607a5b231fd7c2e5f" Oct 31 22:15:06.454257 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.454237 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm_odh-ai-gateway-infra(4bc62938-ef10-4235-91af-d518a045ce95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" podUID="4bc62938-ef10-4235-91af-d518a045ce95" Oct 31 22:15:06.455501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.455485 4294 scope.go:122] "RemoveContainer" containerID="024f3f84fe050b52a5264ca7582a9338173e22c605c461d3167a26582c2ffc33" Oct 31 22:15:06.455734 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.455717 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms_odh-ai-gateway-infra(3ee08211-d158-43a4-a055-61d92d2ee655)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" Oct 31 22:15:06.456908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.456890 4294 scope.go:122] "RemoveContainer" containerID="d680dced1ea449bfd7248b30c5a42f6d47eabc6c5a0e37af6b72f24d2b4fb281" Oct 31 22:15:06.457123 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.457102 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt_odh-ai-gateway-infra(7c519470-3a10-488d-a542-20423d52211a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" podUID="7c519470-3a10-488d-a542-20423d52211a" Oct 31 22:15:06.458494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.458480 4294 scope.go:122] "RemoveContainer" containerID="815fc56b19ac477b19e7e7d68094e320d25df0e1b680d706761786cd844ab15f" Oct 31 22:15:06.458711 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.458692 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt_odh-ai-gateway-infra(4312b8c4-a53c-4354-b696-695e71f214d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" Oct 31 22:15:06.460099 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.460083 4294 scope.go:122] "RemoveContainer" containerID="6c8f7fd456e375208655ccb08a2f07d2a4d8d3c5fd72a770401fa3f62d95a869" Oct 31 22:15:06.460296 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.460274 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4_odh-ai-gateway-infra(845f9969-a52b-4d80-a641-81c2e8020c54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" Oct 31 22:15:06.461583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:06.461568 4294 scope.go:122] "RemoveContainer" containerID="a9e4c772d74698ed9488bafed7708032d26091f93531efa0974115a057b449b3" Oct 31 22:15:06.461754 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:06.461738 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn_odh-ai-gateway-infra(728fa16a-ae89-4220-8990-7ac122157f07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" podUID="728fa16a-ae89-4220-8990-7ac122157f07" Oct 31 22:15:13.987229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:13.987150 4294 scope.go:122] "RemoveContainer" containerID="428749d24b091dd7d13175715d522f991a7beaf9053d59b5e9af81612d4af8b2" Oct 31 22:15:14.526476 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:14.526439 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" event={"ID":"a541b3ad-0cf9-4a2e-b488-f2d54d250c83","Type":"ContainerStarted","Data":"15c50c5e4458d9fc39f4037046e83cac8105927691cf9367379799ea7c26a0b1"} Oct 31 22:15:14.567370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:14.567219 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" podStartSLOduration=14.567201013 podStartE2EDuration="14.567201013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:14.566730865 +0000 UTC m=+3554.691838099" watchObservedRunningTime="2026-10-31 22:15:14.567201013 +0000 UTC m=+3554.692308244" Oct 31 22:15:14.890631 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:14.890593 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms"] Oct 31 22:15:15.179543 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.179511 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:15.298323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.294494 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ee08211-d158-43a4-a055-61d92d2ee655-kube-api-access-b5hjp\" (UniqueName: \"kubernetes.io/projected/3ee08211-d158-43a4-a055-61d92d2ee655-kube-api-access-b5hjp\") pod \"3ee08211-d158-43a4-a055-61d92d2ee655\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " Oct 31 22:15:15.298323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.294722 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ee08211-d158-43a4-a055-61d92d2ee655-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee08211-d158-43a4-a055-61d92d2ee655-maas-api-service-ca\") pod \"3ee08211-d158-43a4-a055-61d92d2ee655\" (UID: \"3ee08211-d158-43a4-a055-61d92d2ee655\") " Oct 31 22:15:15.314320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.311514 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ee08211-d158-43a4-a055-61d92d2ee655-maas-api-service-ca" pod "3ee08211-d158-43a4-a055-61d92d2ee655" (UID: "3ee08211-d158-43a4-a055-61d92d2ee655"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:15.376378 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.375837 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ee08211-d158-43a4-a055-61d92d2ee655-kube-api-access-b5hjp" pod "3ee08211-d158-43a4-a055-61d92d2ee655" (UID: "3ee08211-d158-43a4-a055-61d92d2ee655"). InnerVolumeSpecName "kube-api-access-b5hjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:15.410470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.408424 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hjp\" (UniqueName: \"kubernetes.io/projected/3ee08211-d158-43a4-a055-61d92d2ee655-kube-api-access-b5hjp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:15.410470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.408463 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ee08211-d158-43a4-a055-61d92d2ee655-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:15.502234 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.502187 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" podUID="f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" containerName="main" containerID="cri-o://e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb" gracePeriod=45 Oct 31 22:15:15.538859 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.538833 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" Oct 31 22:15:15.539013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.538859 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms" event={"ID":"3ee08211-d158-43a4-a055-61d92d2ee655","Type":"ContainerDied","Data":"35f95a02bb47eb910d1493bf68822487d1f57c7117f52255e5159b315fb09bdb"} Oct 31 22:15:15.539013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.538924 4294 scope.go:122] "RemoveContainer" containerID="024f3f84fe050b52a5264ca7582a9338173e22c605c461d3167a26582c2ffc33" Oct 31 22:15:15.541102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.541077 4294 generic.go:356] "Generic (PLEG): container finished" podID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerID="15c50c5e4458d9fc39f4037046e83cac8105927691cf9367379799ea7c26a0b1" exitCode=60 Oct 31 22:15:15.541218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.541159 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" event={"ID":"a541b3ad-0cf9-4a2e-b488-f2d54d250c83","Type":"ContainerDied","Data":"15c50c5e4458d9fc39f4037046e83cac8105927691cf9367379799ea7c26a0b1"} Oct 31 22:15:15.541681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.541663 4294 scope.go:122] "RemoveContainer" containerID="15c50c5e4458d9fc39f4037046e83cac8105927691cf9367379799ea7c26a0b1" Oct 31 22:15:15.542041 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:15.542020 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc_odh-ai-gateway-infra(a541b3ad-0cf9-4a2e-b488-f2d54d250c83)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" Oct 31 22:15:15.579034 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.579000 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc"] Oct 31 22:15:15.620529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.620509 4294 scope.go:122] "RemoveContainer" containerID="428749d24b091dd7d13175715d522f991a7beaf9053d59b5e9af81612d4af8b2" Oct 31 22:15:15.721322 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.707756 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" podUID="f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" containerName="main" probeResult="failure" output="Get \"https://10.134.1.46:8000/ready\": dial tcp 10.134.1.46:8000: connect: connection refused" Oct 31 22:15:15.747326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.724364 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms"] Oct 31 22:15:15.832168 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.832134 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a5affe-29891415-4fqms"] Oct 31 22:15:15.855388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.855358 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k"] Oct 31 22:15:15.858852 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:15.858826 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a5affe-rf98k"] Oct 31 22:15:16.024459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.023573 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" path="/var/lib/kubelet/pods/3ee08211-d158-43a4-a055-61d92d2ee655/volumes" Oct 31 22:15:16.025554 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.025528 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa46b9c-6693-49dc-a07d-15ecb0c46dbe" path="/var/lib/kubelet/pods/9aa46b9c-6693-49dc-a07d-15ecb0c46dbe/volumes" Oct 31 22:15:16.026690 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.026672 4294 scope.go:122] "RemoveContainer" containerID="ee98f272dd6963c175343b5fbc1e073ce17cae58ef12aca607a5b231fd7c2e5f" Oct 31 22:15:16.027154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.027109 4294 scope.go:122] "RemoveContainer" containerID="a9e4c772d74698ed9488bafed7708032d26091f93531efa0974115a057b449b3" Oct 31 22:15:16.127942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.127906 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:15:16.258841 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.258792 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-kube-api-access-rx2g6\" (UniqueName: \"kubernetes.io/projected/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-kube-api-access-rx2g6\") pod \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " Oct 31 22:15:16.259401 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.258982 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tls-certs\") pod \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " Oct 31 22:15:16.259401 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.259111 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tmp-dir\") pod \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " Oct 31 22:15:16.259401 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.259220 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-home\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-home\") pod \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " Oct 31 22:15:16.259401 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.259260 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-dshm\") pod \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " Oct 31 22:15:16.259401 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.259291 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-model-cache\") pod \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\" (UID: \"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405\") " Oct 31 22:15:16.259679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.259518 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tmp-dir" pod "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" (UID: "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:16.259679 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.259639 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-home" pod "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" (UID: "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:16.259849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.259818 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-model-cache" pod "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" (UID: "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:16.260419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.260394 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.260507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.260424 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.260507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.260438 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.279893 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.275882 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-kube-api-access-rx2g6" pod "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" (UID: "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405"). InnerVolumeSpecName "kube-api-access-rx2g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:16.279893 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.279799 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-dshm" pod "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" (UID: "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:16.280248 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.280218 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tls-certs" pod "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" (UID: "f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:16.361523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.361482 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.361523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.361520 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx2g6\" (UniqueName: \"kubernetes.io/projected/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-kube-api-access-rx2g6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.361737 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.361537 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.552182 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.552080 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerStarted","Data":"8f5ce8a0010f124f7d89fb4f762dadf876afd6937c12dd7fb610a6b908c24341"} Oct 31 22:15:16.558157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.558129 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" event={"ID":"728fa16a-ae89-4220-8990-7ac122157f07","Type":"ContainerStarted","Data":"30eff3a0df76a35f03fb33a72b1d3bc0947d5fe73687906ee665b24beb972f2b"} Oct 31 22:15:16.559758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.559732 4294 generic.go:356] "Generic (PLEG): container finished" podID="f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" containerID="e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb" exitCode=0 Oct 31 22:15:16.559871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.559762 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" Oct 31 22:15:16.559871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.559782 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" event={"ID":"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405","Type":"ContainerDied","Data":"e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb"} Oct 31 22:15:16.559871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.559828 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp" event={"ID":"f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405","Type":"ContainerDied","Data":"4fb2d110053608f34b44b033a3b5b4696530a0833de7a5dc7854c37421e82d43"} Oct 31 22:15:16.559871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.559851 4294 scope.go:122] "RemoveContainer" containerID="e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb" Oct 31 22:15:16.573522 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.573502 4294 scope.go:122] "RemoveContainer" containerID="e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb" Oct 31 22:15:16.573896 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:16.573867 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb\": container with ID starting with e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb not found: ID does not exist" containerID="e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb" Oct 31 22:15:16.573982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.573902 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb"} err="failed to get container status \"e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb\": rpc error: code = NotFound desc = could not find container \"e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb\": container with ID starting with e6d7e673e3d3baf04b799fa0d22b8342815434f71736919b865e96df7838a5bb not found: ID does not exist" Oct 31 22:15:16.747339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.734560 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp"] Oct 31 22:15:16.747339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.745449 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-710927/test-model-710927-kserve-574cf84d79-kwpvp"] Oct 31 22:15:16.769777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.769742 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:16.870706 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.870611 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-kube-api-access-qz5td\" (UniqueName: \"kubernetes.io/projected/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-kube-api-access-qz5td\") pod \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " Oct 31 22:15:16.870880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.870732 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-maas-api-service-ca\") pod \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\" (UID: \"a541b3ad-0cf9-4a2e-b488-f2d54d250c83\") " Oct 31 22:15:16.871143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.871103 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-maas-api-service-ca" pod "a541b3ad-0cf9-4a2e-b488-f2d54d250c83" (UID: "a541b3ad-0cf9-4a2e-b488-f2d54d250c83"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:16.871414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.871394 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.873294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.873266 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-kube-api-access-qz5td" pod "a541b3ad-0cf9-4a2e-b488-f2d54d250c83" (UID: "a541b3ad-0cf9-4a2e-b488-f2d54d250c83"). InnerVolumeSpecName "kube-api-access-qz5td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:16.973132 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.973089 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz5td\" (UniqueName: \"kubernetes.io/projected/a541b3ad-0cf9-4a2e-b488-f2d54d250c83-kube-api-access-qz5td\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:16.978189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:16.978161 4294 scope.go:122] "RemoveContainer" containerID="815fc56b19ac477b19e7e7d68094e320d25df0e1b680d706761786cd844ab15f" Oct 31 22:15:17.335498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.335463 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl"] Oct 31 22:15:17.336559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336535 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" containerName="cleanup" Oct 31 22:15:17.336559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336557 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" containerName="cleanup" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336587 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336593 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336609 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" containerName="cleanup" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336613 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" containerName="cleanup" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336636 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336641 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336660 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" containerName="main" Oct 31 22:15:17.336701 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.336664 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" containerName="main" Oct 31 22:15:17.341929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.340421 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" containerName="cleanup" Oct 31 22:15:17.341929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.340475 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:17.341929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.340488 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:17.341929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.340511 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" containerName="main" Oct 31 22:15:17.341929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.340539 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:17.350143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.350120 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.362173 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.362143 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl"] Oct 31 22:15:17.372354 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.369634 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:15:17.391650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.391625 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.391934 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.391907 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqqjg\" (UniqueName: \"kubernetes.io/projected/bae7637d-1487-4f9b-acda-cc1a4bd37725-kube-api-access-vqqjg\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.392032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.391956 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.493755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.493708 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.493966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.493947 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqqjg\" (UniqueName: \"kubernetes.io/projected/bae7637d-1487-4f9b-acda-cc1a4bd37725-kube-api-access-vqqjg\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.494013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.493998 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.494435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.494414 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.496674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.496650 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.521319 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.521268 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqqjg\" (UniqueName: \"kubernetes.io/projected/bae7637d-1487-4f9b-acda-cc1a4bd37725-kube-api-access-vqqjg\") pod \"maas-api-revoke-keys-e2e-inf-a-710927-q5rsl\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.571297 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.571258 4294 generic.go:356] "Generic (PLEG): container finished" podID="728fa16a-ae89-4220-8990-7ac122157f07" containerID="30eff3a0df76a35f03fb33a72b1d3bc0947d5fe73687906ee665b24beb972f2b" exitCode=60 Oct 31 22:15:17.571477 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.571337 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" event={"ID":"728fa16a-ae89-4220-8990-7ac122157f07","Type":"ContainerDied","Data":"30eff3a0df76a35f03fb33a72b1d3bc0947d5fe73687906ee665b24beb972f2b"} Oct 31 22:15:17.571477 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.571406 4294 scope.go:122] "RemoveContainer" containerID="a9e4c772d74698ed9488bafed7708032d26091f93531efa0974115a057b449b3" Oct 31 22:15:17.571993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.571971 4294 scope.go:122] "RemoveContainer" containerID="30eff3a0df76a35f03fb33a72b1d3bc0947d5fe73687906ee665b24beb972f2b" Oct 31 22:15:17.572480 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:17.572455 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn_odh-ai-gateway-infra(728fa16a-ae89-4220-8990-7ac122157f07)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" podUID="728fa16a-ae89-4220-8990-7ac122157f07" Oct 31 22:15:17.575889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.575866 4294 generic.go:356] "Generic (PLEG): container finished" podID="4bc62938-ef10-4235-91af-d518a045ce95" containerID="8f5ce8a0010f124f7d89fb4f762dadf876afd6937c12dd7fb610a6b908c24341" exitCode=60 Oct 31 22:15:17.576025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.576000 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerDied","Data":"8f5ce8a0010f124f7d89fb4f762dadf876afd6937c12dd7fb610a6b908c24341"} Oct 31 22:15:17.576470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.576444 4294 scope.go:122] "RemoveContainer" containerID="8f5ce8a0010f124f7d89fb4f762dadf876afd6937c12dd7fb610a6b908c24341" Oct 31 22:15:17.576859 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:17.576837 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm_odh-ai-gateway-infra(4bc62938-ef10-4235-91af-d518a045ce95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" podUID="4bc62938-ef10-4235-91af-d518a045ce95" Oct 31 22:15:17.578892 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.578867 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerStarted","Data":"8880e5cd0d7d2b1491fae389695b34e171994596c66d25e6d20b41772280873b"} Oct 31 22:15:17.580357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.580331 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" event={"ID":"a541b3ad-0cf9-4a2e-b488-f2d54d250c83","Type":"ContainerDied","Data":"462da6e4acbe1f3204662d78b1a8123e1b4f8f6e3ef9566d2c56f0b3ca132ab6"} Oct 31 22:15:17.580461 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.580380 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc" Oct 31 22:15:17.588584 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.588564 4294 scope.go:122] "RemoveContainer" containerID="ee98f272dd6963c175343b5fbc1e073ce17cae58ef12aca607a5b231fd7c2e5f" Oct 31 22:15:17.604395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.604368 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm"] Oct 31 22:15:17.616781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.616755 4294 scope.go:122] "RemoveContainer" containerID="15c50c5e4458d9fc39f4037046e83cac8105927691cf9367379799ea7c26a0b1" Oct 31 22:15:17.672595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.672561 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:17.705843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.705808 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn"] Oct 31 22:15:17.739319 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.739266 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc"] Oct 31 22:15:17.761351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.752561 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-44405d-29891415-pthvc"] Oct 31 22:15:17.987278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.987246 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" path="/var/lib/kubelet/pods/a541b3ad-0cf9-4a2e-b488-f2d54d250c83/volumes" Oct 31 22:15:17.987838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.987815 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405" path="/var/lib/kubelet/pods/f9bb315b-5ef8-4c5c-8eb3-79ea3e14b405/volumes" Oct 31 22:15:17.988843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.988725 4294 scope.go:122] "RemoveContainer" containerID="01d602db5500c43ec5ef3352720bd1cd5bc29c14ea3ec671397497df9d081b65" Oct 31 22:15:17.988966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.988929 4294 scope.go:122] "RemoveContainer" containerID="d680dced1ea449bfd7248b30c5a42f6d47eabc6c5a0e37af6b72f24d2b4fb281" Oct 31 22:15:17.989092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.989075 4294 scope.go:122] "RemoveContainer" containerID="bcc322a72b8d5c81f85a9676212dd8915bf390ab72ff2dfc60647d34ee8c6bb9" Oct 31 22:15:17.989565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.989546 4294 scope.go:122] "RemoveContainer" containerID="98823e9c6baa0d1295de03bed96254125a1a67a6481e2b435717faef81145e1a" Oct 31 22:15:17.991586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:17.991563 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl"] Oct 31 22:15:17.992611 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:17.992589 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae7637d_1487_4f9b_acda_cc1a4bd37725.slice/crio-5f6e7109060f166825c9ae8d86c6de4773988050ed542d6775de045daec94c30 WatchSource:0}: Error finding container 5f6e7109060f166825c9ae8d86c6de4773988050ed542d6775de045daec94c30: Status 404 returned error can't find the container with id 5f6e7109060f166825c9ae8d86c6de4773988050ed542d6775de045daec94c30 Oct 31 22:15:18.592440 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.592413 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" event={"ID":"bae7637d-1487-4f9b-acda-cc1a4bd37725","Type":"ContainerStarted","Data":"5f6e7109060f166825c9ae8d86c6de4773988050ed542d6775de045daec94c30"} Oct 31 22:15:18.596383 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.596344 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" event={"ID":"31db38fe-c0de-4701-b7ed-62b8a7539a1a","Type":"ContainerStarted","Data":"f970bcda1882f5a726ff7b79146cb253757ee00cf89d6761faa0949cf9d44ae7"} Oct 31 22:15:18.598333 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.598280 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerStarted","Data":"15952c111596f9b61fd4975b0e619c8ad9eba3926c33277fc1edb11e8d2e827c"} Oct 31 22:15:18.603601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.603581 4294 generic.go:356] "Generic (PLEG): container finished" podID="4312b8c4-a53c-4354-b696-695e71f214d6" containerID="8880e5cd0d7d2b1491fae389695b34e171994596c66d25e6d20b41772280873b" exitCode=60 Oct 31 22:15:18.603682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.603617 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerDied","Data":"8880e5cd0d7d2b1491fae389695b34e171994596c66d25e6d20b41772280873b"} Oct 31 22:15:18.603682 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.603661 4294 scope.go:122] "RemoveContainer" containerID="815fc56b19ac477b19e7e7d68094e320d25df0e1b680d706761786cd844ab15f" Oct 31 22:15:18.604013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.603994 4294 scope.go:122] "RemoveContainer" containerID="8880e5cd0d7d2b1491fae389695b34e171994596c66d25e6d20b41772280873b" Oct 31 22:15:18.604383 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:18.604353 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt_odh-ai-gateway-infra(4312b8c4-a53c-4354-b696-695e71f214d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" Oct 31 22:15:18.667161 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.667134 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt"] Oct 31 22:15:18.904067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.903990 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:18.917044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.917009 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728fa16a-ae89-4220-8990-7ac122157f07-kube-api-access-hsmn9\" (UniqueName: \"kubernetes.io/projected/728fa16a-ae89-4220-8990-7ac122157f07-kube-api-access-hsmn9\") pod \"728fa16a-ae89-4220-8990-7ac122157f07\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " Oct 31 22:15:18.917164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.917127 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728fa16a-ae89-4220-8990-7ac122157f07-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728fa16a-ae89-4220-8990-7ac122157f07-maas-api-service-ca\") pod \"728fa16a-ae89-4220-8990-7ac122157f07\" (UID: \"728fa16a-ae89-4220-8990-7ac122157f07\") " Oct 31 22:15:18.917776 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.917613 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728fa16a-ae89-4220-8990-7ac122157f07-maas-api-service-ca" pod "728fa16a-ae89-4220-8990-7ac122157f07" (UID: "728fa16a-ae89-4220-8990-7ac122157f07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:18.917924 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.917905 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/728fa16a-ae89-4220-8990-7ac122157f07-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:18.919088 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.919027 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728fa16a-ae89-4220-8990-7ac122157f07-kube-api-access-hsmn9" pod "728fa16a-ae89-4220-8990-7ac122157f07" (UID: "728fa16a-ae89-4220-8990-7ac122157f07"). InnerVolumeSpecName "kube-api-access-hsmn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:18.920180 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.920164 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:18.979531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.979496 4294 scope.go:122] "RemoveContainer" containerID="be8083c7f06b3a3c0074649da93e0ca57954a2847f493357c6d10107bd50a036" Oct 31 22:15:18.984700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.984659 4294 scope.go:122] "RemoveContainer" containerID="48ff12bdf699909d80879b04e9b89a8a101775e05c2351781304ecd192df4a40" Oct 31 22:15:18.985067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.985047 4294 scope.go:122] "RemoveContainer" containerID="6c8f7fd456e375208655ccb08a2f07d2a4d8d3c5fd72a770401fa3f62d95a869" Oct 31 22:15:18.985162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.985133 4294 scope.go:122] "RemoveContainer" containerID="40e750eef599b5fd4d075a7fedc20fe3511dc71ee5ad49a09fb4b994e52f8f41" Oct 31 22:15:18.985252 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:18.985235 4294 scope.go:122] "RemoveContainer" containerID="48ff77df7eb4efae05fee31bef733405dd166ba3ae4e7711aaf581f170a21719" Oct 31 22:15:19.019355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.018517 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc62938-ef10-4235-91af-d518a045ce95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc62938-ef10-4235-91af-d518a045ce95-maas-api-service-ca\") pod \"4bc62938-ef10-4235-91af-d518a045ce95\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " Oct 31 22:15:19.019355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.018834 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc62938-ef10-4235-91af-d518a045ce95-kube-api-access-zxt4k\" (UniqueName: \"kubernetes.io/projected/4bc62938-ef10-4235-91af-d518a045ce95-kube-api-access-zxt4k\") pod \"4bc62938-ef10-4235-91af-d518a045ce95\" (UID: \"4bc62938-ef10-4235-91af-d518a045ce95\") " Oct 31 22:15:19.019657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.019620 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmn9\" (UniqueName: \"kubernetes.io/projected/728fa16a-ae89-4220-8990-7ac122157f07-kube-api-access-hsmn9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:19.020324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.020268 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc62938-ef10-4235-91af-d518a045ce95-maas-api-service-ca" pod "4bc62938-ef10-4235-91af-d518a045ce95" (UID: "4bc62938-ef10-4235-91af-d518a045ce95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:19.025513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.025457 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc62938-ef10-4235-91af-d518a045ce95-kube-api-access-zxt4k" pod "4bc62938-ef10-4235-91af-d518a045ce95" (UID: "4bc62938-ef10-4235-91af-d518a045ce95"). InnerVolumeSpecName "kube-api-access-zxt4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:19.121177 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.121101 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxt4k\" (UniqueName: \"kubernetes.io/projected/4bc62938-ef10-4235-91af-d518a045ce95-kube-api-access-zxt4k\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:19.121177 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.121127 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bc62938-ef10-4235-91af-d518a045ce95-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:19.624436 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.624398 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" event={"ID":"4bc62938-ef10-4235-91af-d518a045ce95","Type":"ContainerDied","Data":"90da6f35e773317e16b6f8c9a471550b38ea92083474183958d42626eb6200cd"} Oct 31 22:15:19.624977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.624477 4294 scope.go:122] "RemoveContainer" containerID="8f5ce8a0010f124f7d89fb4f762dadf876afd6937c12dd7fb610a6b908c24341" Oct 31 22:15:19.624977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.624511 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm" Oct 31 22:15:19.629597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.629539 4294 generic.go:356] "Generic (PLEG): container finished" podID="7c519470-3a10-488d-a542-20423d52211a" containerID="d59b7daf04b0a07beafc4e53e116ddbf1b025cac27cb8c9c00e444bf467a8edf" exitCode=60 Oct 31 22:15:19.629738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.629699 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" event={"ID":"7c519470-3a10-488d-a542-20423d52211a","Type":"ContainerDied","Data":"d59b7daf04b0a07beafc4e53e116ddbf1b025cac27cb8c9c00e444bf467a8edf"} Oct 31 22:15:19.630390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.630366 4294 scope.go:122] "RemoveContainer" containerID="d59b7daf04b0a07beafc4e53e116ddbf1b025cac27cb8c9c00e444bf467a8edf" Oct 31 22:15:19.630664 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:19.630639 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt_odh-ai-gateway-infra(7c519470-3a10-488d-a542-20423d52211a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" podUID="7c519470-3a10-488d-a542-20423d52211a" Oct 31 22:15:19.636584 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.636133 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerStarted","Data":"3e158098f4b00f978812e38d4cd55e7b1bc4a792cc68d307d9f5f81e0bc0c03e"} Oct 31 22:15:19.638474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.638432 4294 generic.go:356] "Generic (PLEG): container finished" podID="bae7637d-1487-4f9b-acda-cc1a4bd37725" containerID="bc16856dd3160949babdf9d09a88b6e97e85fea72519fc46d0e34eda5d69bf88" exitCode=0 Oct 31 22:15:19.638604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.638567 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" event={"ID":"bae7637d-1487-4f9b-acda-cc1a4bd37725","Type":"ContainerDied","Data":"bc16856dd3160949babdf9d09a88b6e97e85fea72519fc46d0e34eda5d69bf88"} Oct 31 22:15:19.640973 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.640905 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" event={"ID":"728fa16a-ae89-4220-8990-7ac122157f07","Type":"ContainerDied","Data":"5273aa262c0a4e5c13b26c802a8baf546e0c4b162bdd79f8ce4fd89f58562ebb"} Oct 31 22:15:19.642424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.642400 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn" Oct 31 22:15:19.644267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.644244 4294 generic.go:356] "Generic (PLEG): container finished" podID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerID="3dd2c5c6001ccab235976f3a477ada73f91f6cedb50dd9e5a3f9c9e64e3cae1f" exitCode=60 Oct 31 22:15:19.644370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.644324 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" event={"ID":"f6e5c17f-0945-4321-b1d4-26c06bfd778c","Type":"ContainerDied","Data":"3dd2c5c6001ccab235976f3a477ada73f91f6cedb50dd9e5a3f9c9e64e3cae1f"} Oct 31 22:15:19.644747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.644726 4294 scope.go:122] "RemoveContainer" containerID="3dd2c5c6001ccab235976f3a477ada73f91f6cedb50dd9e5a3f9c9e64e3cae1f" Oct 31 22:15:19.645070 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:19.645051 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp_odh-ai-gateway-infra(f6e5c17f-0945-4321-b1d4-26c06bfd778c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" Oct 31 22:15:19.645974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.645952 4294 scope.go:122] "RemoveContainer" containerID="d680dced1ea449bfd7248b30c5a42f6d47eabc6c5a0e37af6b72f24d2b4fb281" Oct 31 22:15:19.646442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.646419 4294 generic.go:356] "Generic (PLEG): container finished" podID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerID="f970bcda1882f5a726ff7b79146cb253757ee00cf89d6761faa0949cf9d44ae7" exitCode=60 Oct 31 22:15:19.646538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.646490 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" event={"ID":"31db38fe-c0de-4701-b7ed-62b8a7539a1a","Type":"ContainerDied","Data":"f970bcda1882f5a726ff7b79146cb253757ee00cf89d6761faa0949cf9d44ae7"} Oct 31 22:15:19.646884 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.646867 4294 scope.go:122] "RemoveContainer" containerID="f970bcda1882f5a726ff7b79146cb253757ee00cf89d6761faa0949cf9d44ae7" Oct 31 22:15:19.647206 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:19.647178 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz_odh-ai-gateway-infra(31db38fe-c0de-4701-b7ed-62b8a7539a1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" Oct 31 22:15:19.648806 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.648773 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" event={"ID":"3aa3eb2b-4e44-4796-904f-a9aec7047555","Type":"ContainerStarted","Data":"d63d61b7c33f4d52ff77be5cc545abe0387dcdfd11f57d30e2e30b40c5e918ef"} Oct 31 22:15:19.651021 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.650986 4294 generic.go:356] "Generic (PLEG): container finished" podID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerID="15952c111596f9b61fd4975b0e619c8ad9eba3926c33277fc1edb11e8d2e827c" exitCode=60 Oct 31 22:15:19.651146 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.651026 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerDied","Data":"15952c111596f9b61fd4975b0e619c8ad9eba3926c33277fc1edb11e8d2e827c"} Oct 31 22:15:19.651442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.651422 4294 scope.go:122] "RemoveContainer" containerID="15952c111596f9b61fd4975b0e619c8ad9eba3926c33277fc1edb11e8d2e827c" Oct 31 22:15:19.651685 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:19.651665 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd_odh-ai-gateway-infra(d7a1dfb1-a6f3-42fa-8566-bb614078045a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" Oct 31 22:15:19.654463 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.654436 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerStarted","Data":"0dd9c1ef258ef650b1c30da84143db4cf1b1bd769cb89e80b618e687b41b0f5b"} Oct 31 22:15:19.669180 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.669140 4294 scope.go:122] "RemoveContainer" containerID="30eff3a0df76a35f03fb33a72b1d3bc0947d5fe73687906ee665b24beb972f2b" Oct 31 22:15:19.685678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.685652 4294 scope.go:122] "RemoveContainer" containerID="bcc322a72b8d5c81f85a9676212dd8915bf390ab72ff2dfc60647d34ee8c6bb9" Oct 31 22:15:19.701554 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.701533 4294 scope.go:122] "RemoveContainer" containerID="98823e9c6baa0d1295de03bed96254125a1a67a6481e2b435717faef81145e1a" Oct 31 22:15:19.725575 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.725541 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz"] Oct 31 22:15:19.730814 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.730787 4294 scope.go:122] "RemoveContainer" containerID="01d602db5500c43ec5ef3352720bd1cd5bc29c14ea3ec671397497df9d081b65" Oct 31 22:15:19.776697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.776663 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd"] Oct 31 22:15:19.816132 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.816106 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:19.832721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.832695 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4312b8c4-a53c-4354-b696-695e71f214d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4312b8c4-a53c-4354-b696-695e71f214d6-maas-api-service-ca\") pod \"4312b8c4-a53c-4354-b696-695e71f214d6\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " Oct 31 22:15:19.833018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.833000 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4312b8c4-a53c-4354-b696-695e71f214d6-kube-api-access-vj5ml\" (UniqueName: \"kubernetes.io/projected/4312b8c4-a53c-4354-b696-695e71f214d6-kube-api-access-vj5ml\") pod \"4312b8c4-a53c-4354-b696-695e71f214d6\" (UID: \"4312b8c4-a53c-4354-b696-695e71f214d6\") " Oct 31 22:15:19.833101 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.833049 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4312b8c4-a53c-4354-b696-695e71f214d6-maas-api-service-ca" pod "4312b8c4-a53c-4354-b696-695e71f214d6" (UID: "4312b8c4-a53c-4354-b696-695e71f214d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:19.833466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.833450 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4312b8c4-a53c-4354-b696-695e71f214d6-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:19.835047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.835025 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4312b8c4-a53c-4354-b696-695e71f214d6-kube-api-access-vj5ml" pod "4312b8c4-a53c-4354-b696-695e71f214d6" (UID: "4312b8c4-a53c-4354-b696-695e71f214d6"). InnerVolumeSpecName "kube-api-access-vj5ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:19.844993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.844967 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm"] Oct 31 22:15:19.851505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.851482 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-fb32d4-29891415-l8tjm"] Oct 31 22:15:19.934657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.934574 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj5ml\" (UniqueName: \"kubernetes.io/projected/4312b8c4-a53c-4354-b696-695e71f214d6-kube-api-access-vj5ml\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:19.965357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.965323 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn"] Oct 31 22:15:19.986011 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.985979 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc62938-ef10-4235-91af-d518a045ce95" path="/var/lib/kubelet/pods/4bc62938-ef10-4235-91af-d518a045ce95/volumes" Oct 31 22:15:19.986695 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:19.986671 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-6b5dab-29891415-p7crn"] Oct 31 22:15:20.664664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.664580 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" Oct 31 22:15:20.664664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.664593 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt" event={"ID":"4312b8c4-a53c-4354-b696-695e71f214d6","Type":"ContainerDied","Data":"d54c0ab4b7d599e152054f488c80536ddf251c3171aaebc9ec799d00aa288bea"} Oct 31 22:15:20.664664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.664659 4294 scope.go:122] "RemoveContainer" containerID="8880e5cd0d7d2b1491fae389695b34e171994596c66d25e6d20b41772280873b" Oct 31 22:15:20.666672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.666644 4294 generic.go:356] "Generic (PLEG): container finished" podID="845f9969-a52b-4d80-a641-81c2e8020c54" containerID="3e158098f4b00f978812e38d4cd55e7b1bc4a792cc68d307d9f5f81e0bc0c03e" exitCode=60 Oct 31 22:15:20.666822 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.666726 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerDied","Data":"3e158098f4b00f978812e38d4cd55e7b1bc4a792cc68d307d9f5f81e0bc0c03e"} Oct 31 22:15:20.667691 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.667153 4294 scope.go:122] "RemoveContainer" containerID="3e158098f4b00f978812e38d4cd55e7b1bc4a792cc68d307d9f5f81e0bc0c03e" Oct 31 22:15:20.667691 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:20.667502 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4_odh-ai-gateway-infra(845f9969-a52b-4d80-a641-81c2e8020c54)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" Oct 31 22:15:20.669944 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.669921 4294 generic.go:356] "Generic (PLEG): container finished" podID="9f0741c6-c011-4c39-b014-0d9034978b03" containerID="ebf373eb9f65adbb6c20e2629dd26acab97d76d4466578498e984fd194cbff83" exitCode=60 Oct 31 22:15:20.670049 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.669989 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" event={"ID":"9f0741c6-c011-4c39-b014-0d9034978b03","Type":"ContainerDied","Data":"ebf373eb9f65adbb6c20e2629dd26acab97d76d4466578498e984fd194cbff83"} Oct 31 22:15:20.670394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.670378 4294 scope.go:122] "RemoveContainer" containerID="ebf373eb9f65adbb6c20e2629dd26acab97d76d4466578498e984fd194cbff83" Oct 31 22:15:20.670665 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:20.670645 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw_odh-ai-gateway-infra(9f0741c6-c011-4c39-b014-0d9034978b03)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" Oct 31 22:15:20.679490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.679472 4294 scope.go:122] "RemoveContainer" containerID="6c8f7fd456e375208655ccb08a2f07d2a4d8d3c5fd72a770401fa3f62d95a869" Oct 31 22:15:20.680007 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.679988 4294 generic.go:356] "Generic (PLEG): container finished" podID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerID="d63d61b7c33f4d52ff77be5cc545abe0387dcdfd11f57d30e2e30b40c5e918ef" exitCode=60 Oct 31 22:15:20.680090 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.680064 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" event={"ID":"3aa3eb2b-4e44-4796-904f-a9aec7047555","Type":"ContainerDied","Data":"d63d61b7c33f4d52ff77be5cc545abe0387dcdfd11f57d30e2e30b40c5e918ef"} Oct 31 22:15:20.680532 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.680510 4294 scope.go:122] "RemoveContainer" containerID="d63d61b7c33f4d52ff77be5cc545abe0387dcdfd11f57d30e2e30b40c5e918ef" Oct 31 22:15:20.680903 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:20.680875 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv_odh-ai-gateway-infra(3aa3eb2b-4e44-4796-904f-a9aec7047555)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" Oct 31 22:15:20.682089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.682068 4294 generic.go:356] "Generic (PLEG): container finished" podID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerID="91b3d95c412b84ed26ac173e823e23d17b78ec675864d669ddcd00e1b950c5a7" exitCode=60 Oct 31 22:15:20.682194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.682140 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" event={"ID":"8db9dfad-1040-4e7c-8203-1497b21954f0","Type":"ContainerDied","Data":"91b3d95c412b84ed26ac173e823e23d17b78ec675864d669ddcd00e1b950c5a7"} Oct 31 22:15:20.682556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.682539 4294 scope.go:122] "RemoveContainer" containerID="91b3d95c412b84ed26ac173e823e23d17b78ec675864d669ddcd00e1b950c5a7" Oct 31 22:15:20.682845 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:20.682821 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn_odh-ai-gateway-infra(8db9dfad-1040-4e7c-8203-1497b21954f0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" Oct 31 22:15:20.685635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.685610 4294 generic.go:356] "Generic (PLEG): container finished" podID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerID="0dd9c1ef258ef650b1c30da84143db4cf1b1bd769cb89e80b618e687b41b0f5b" exitCode=60 Oct 31 22:15:20.685734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.685714 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerDied","Data":"0dd9c1ef258ef650b1c30da84143db4cf1b1bd769cb89e80b618e687b41b0f5b"} Oct 31 22:15:20.686098 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.686080 4294 scope.go:122] "RemoveContainer" containerID="0dd9c1ef258ef650b1c30da84143db4cf1b1bd769cb89e80b618e687b41b0f5b" Oct 31 22:15:20.686436 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:20.686412 4294 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g_odh-ai-gateway-infra(d5a772d8-9d68-4a2f-a241-6f356eb5a2c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" Oct 31 22:15:20.695142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.695111 4294 scope.go:122] "RemoveContainer" containerID="40e750eef599b5fd4d075a7fedc20fe3511dc71ee5ad49a09fb4b994e52f8f41" Oct 31 22:15:20.715171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.714955 4294 scope.go:122] "RemoveContainer" containerID="48ff12bdf699909d80879b04e9b89a8a101775e05c2351781304ecd192df4a40" Oct 31 22:15:20.723035 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.722981 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt"] Oct 31 22:15:20.734329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.731361 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-c34cbd-29891415-6bbpt"] Oct 31 22:15:20.739693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.739662 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt"] Oct 31 22:15:20.743824 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.743803 4294 scope.go:122] "RemoveContainer" containerID="48ff77df7eb4efae05fee31bef733405dd166ba3ae4e7711aaf581f170a21719" Oct 31 22:15:20.760917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.760879 4294 scope.go:122] "RemoveContainer" containerID="be8083c7f06b3a3c0074649da93e0ca57954a2847f493357c6d10107bd50a036" Oct 31 22:15:20.809493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.804784 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4"] Oct 31 22:15:20.881816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.881794 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:20.905160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.905124 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp"] Oct 31 22:15:20.935678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.935647 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:20.969693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.960921 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31db38fe-c0de-4701-b7ed-62b8a7539a1a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31db38fe-c0de-4701-b7ed-62b8a7539a1a-maas-api-service-ca\") pod \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " Oct 31 22:15:20.969693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.961086 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31db38fe-c0de-4701-b7ed-62b8a7539a1a-kube-api-access-6gdcz\" (UniqueName: \"kubernetes.io/projected/31db38fe-c0de-4701-b7ed-62b8a7539a1a-kube-api-access-6gdcz\") pod \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\" (UID: \"31db38fe-c0de-4701-b7ed-62b8a7539a1a\") " Oct 31 22:15:20.969693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.965177 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31db38fe-c0de-4701-b7ed-62b8a7539a1a-maas-api-service-ca" pod "31db38fe-c0de-4701-b7ed-62b8a7539a1a" (UID: "31db38fe-c0de-4701-b7ed-62b8a7539a1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:20.969693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:20.965987 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31db38fe-c0de-4701-b7ed-62b8a7539a1a-kube-api-access-6gdcz" pod "31db38fe-c0de-4701-b7ed-62b8a7539a1a" (UID: "31db38fe-c0de-4701-b7ed-62b8a7539a1a"). InnerVolumeSpecName "kube-api-access-6gdcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:21.014570 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.014541 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:21.016623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.016601 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:21.062136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062101 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c519470-3a10-488d-a542-20423d52211a-kube-api-access-49vkd\" (UniqueName: \"kubernetes.io/projected/7c519470-3a10-488d-a542-20423d52211a-kube-api-access-49vkd\") pod \"7c519470-3a10-488d-a542-20423d52211a\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " Oct 31 22:15:21.062336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062228 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c519470-3a10-488d-a542-20423d52211a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c519470-3a10-488d-a542-20423d52211a-maas-api-service-ca\") pod \"7c519470-3a10-488d-a542-20423d52211a\" (UID: \"7c519470-3a10-488d-a542-20423d52211a\") " Oct 31 22:15:21.062336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062259 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a1dfb1-a6f3-42fa-8566-bb614078045a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a1dfb1-a6f3-42fa-8566-bb614078045a-maas-api-service-ca\") pod \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " Oct 31 22:15:21.062336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062332 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-xks-ca\") pod \"bae7637d-1487-4f9b-acda-cc1a4bd37725\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " Oct 31 22:15:21.062536 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062467 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae7637d-1487-4f9b-acda-cc1a4bd37725-kube-api-access-vqqjg\" (UniqueName: \"kubernetes.io/projected/bae7637d-1487-4f9b-acda-cc1a4bd37725-kube-api-access-vqqjg\") pod \"bae7637d-1487-4f9b-acda-cc1a4bd37725\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " Oct 31 22:15:21.062536 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062503 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-openshift-ca\") pod \"bae7637d-1487-4f9b-acda-cc1a4bd37725\" (UID: \"bae7637d-1487-4f9b-acda-cc1a4bd37725\") " Oct 31 22:15:21.062651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062566 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a1dfb1-a6f3-42fa-8566-bb614078045a-kube-api-access-56php\" (UniqueName: \"kubernetes.io/projected/d7a1dfb1-a6f3-42fa-8566-bb614078045a-kube-api-access-56php\") pod \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\" (UID: \"d7a1dfb1-a6f3-42fa-8566-bb614078045a\") " Oct 31 22:15:21.062769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062747 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a1dfb1-a6f3-42fa-8566-bb614078045a-maas-api-service-ca" pod "d7a1dfb1-a6f3-42fa-8566-bb614078045a" (UID: "d7a1dfb1-a6f3-42fa-8566-bb614078045a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:21.062828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.062743 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c519470-3a10-488d-a542-20423d52211a-maas-api-service-ca" pod "7c519470-3a10-488d-a542-20423d52211a" (UID: "7c519470-3a10-488d-a542-20423d52211a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:21.063930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.063901 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c519470-3a10-488d-a542-20423d52211a-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.064062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.063935 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7a1dfb1-a6f3-42fa-8566-bb614078045a-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.064062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.063952 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31db38fe-c0de-4701-b7ed-62b8a7539a1a-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.064062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.063970 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gdcz\" (UniqueName: \"kubernetes.io/projected/31db38fe-c0de-4701-b7ed-62b8a7539a1a-kube-api-access-6gdcz\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.064610 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.064585 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-openshift-ca" pod "bae7637d-1487-4f9b-acda-cc1a4bd37725" (UID: "bae7637d-1487-4f9b-acda-cc1a4bd37725"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:21.065917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.065890 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae7637d-1487-4f9b-acda-cc1a4bd37725-kube-api-access-vqqjg" pod "bae7637d-1487-4f9b-acda-cc1a4bd37725" (UID: "bae7637d-1487-4f9b-acda-cc1a4bd37725"). InnerVolumeSpecName "kube-api-access-vqqjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:21.066700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.066672 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-xks-ca" pod "bae7637d-1487-4f9b-acda-cc1a4bd37725" (UID: "bae7637d-1487-4f9b-acda-cc1a4bd37725"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:21.066785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.066719 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c519470-3a10-488d-a542-20423d52211a-kube-api-access-49vkd" pod "7c519470-3a10-488d-a542-20423d52211a" (UID: "7c519470-3a10-488d-a542-20423d52211a"). InnerVolumeSpecName "kube-api-access-49vkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:21.066785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.066760 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a1dfb1-a6f3-42fa-8566-bb614078045a-kube-api-access-56php" pod "d7a1dfb1-a6f3-42fa-8566-bb614078045a" (UID: "d7a1dfb1-a6f3-42fa-8566-bb614078045a"). InnerVolumeSpecName "kube-api-access-56php". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:21.067849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.067825 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv"] Oct 31 22:15:21.093205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.093181 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:21.107276 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.107247 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g"] Oct 31 22:15:21.165527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.165498 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e5c17f-0945-4321-b1d4-26c06bfd778c-kube-api-access-6jxd7\" (UniqueName: \"kubernetes.io/projected/f6e5c17f-0945-4321-b1d4-26c06bfd778c-kube-api-access-6jxd7\") pod \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " Oct 31 22:15:21.165661 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.165563 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e5c17f-0945-4321-b1d4-26c06bfd778c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e5c17f-0945-4321-b1d4-26c06bfd778c-maas-api-service-ca\") pod \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\" (UID: \"f6e5c17f-0945-4321-b1d4-26c06bfd778c\") " Oct 31 22:15:21.166030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.166005 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e5c17f-0945-4321-b1d4-26c06bfd778c-maas-api-service-ca" pod "f6e5c17f-0945-4321-b1d4-26c06bfd778c" (UID: "f6e5c17f-0945-4321-b1d4-26c06bfd778c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:21.166030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.166028 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqqjg\" (UniqueName: \"kubernetes.io/projected/bae7637d-1487-4f9b-acda-cc1a4bd37725-kube-api-access-vqqjg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.166206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.166045 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.166206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.166055 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56php\" (UniqueName: \"kubernetes.io/projected/d7a1dfb1-a6f3-42fa-8566-bb614078045a-kube-api-access-56php\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.166206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.166064 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49vkd\" (UniqueName: \"kubernetes.io/projected/7c519470-3a10-488d-a542-20423d52211a-kube-api-access-49vkd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.166206 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.166073 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bae7637d-1487-4f9b-acda-cc1a4bd37725-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.167927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.167867 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e5c17f-0945-4321-b1d4-26c06bfd778c-kube-api-access-6jxd7" pod "f6e5c17f-0945-4321-b1d4-26c06bfd778c" (UID: "f6e5c17f-0945-4321-b1d4-26c06bfd778c"). InnerVolumeSpecName "kube-api-access-6jxd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:21.266999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.266962 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jxd7\" (UniqueName: \"kubernetes.io/projected/f6e5c17f-0945-4321-b1d4-26c06bfd778c-kube-api-access-6jxd7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.266999 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.266996 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e5c17f-0945-4321-b1d4-26c06bfd778c-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.700674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.700636 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" event={"ID":"f6e5c17f-0945-4321-b1d4-26c06bfd778c","Type":"ContainerDied","Data":"6acbedb3b21585dbdf7fb2a12078b291c22f6f7e3e0a0d8437512797eae4bb86"} Oct 31 22:15:21.701160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.700708 4294 scope.go:122] "RemoveContainer" containerID="3dd2c5c6001ccab235976f3a477ada73f91f6cedb50dd9e5a3f9c9e64e3cae1f" Oct 31 22:15:21.701160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.700645 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp" Oct 31 22:15:21.702228 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.702207 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" Oct 31 22:15:21.702408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.702203 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz" event={"ID":"31db38fe-c0de-4701-b7ed-62b8a7539a1a","Type":"ContainerDied","Data":"cc08b4d7149ff4676269e8dcf01578f124f15a08b64b571a39eff598488674ab"} Oct 31 22:15:21.709164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.709139 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" Oct 31 22:15:21.709331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.709187 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd" event={"ID":"d7a1dfb1-a6f3-42fa-8566-bb614078045a","Type":"ContainerDied","Data":"b0a8d5e81705252b36fdaba40393c29a63375448b5da614967ada26e505cb9bb"} Oct 31 22:15:21.714115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.714092 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" event={"ID":"7c519470-3a10-488d-a542-20423d52211a","Type":"ContainerDied","Data":"87aa8775bac7a50f5c663a731d7e9e1290984127a0591c1910a10336e869469c"} Oct 31 22:15:21.714228 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.714180 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt" Oct 31 22:15:21.715208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.715186 4294 scope.go:122] "RemoveContainer" containerID="f970bcda1882f5a726ff7b79146cb253757ee00cf89d6761faa0949cf9d44ae7" Oct 31 22:15:21.723400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.723375 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" event={"ID":"bae7637d-1487-4f9b-acda-cc1a4bd37725","Type":"ContainerDied","Data":"5f6e7109060f166825c9ae8d86c6de4773988050ed542d6775de045daec94c30"} Oct 31 22:15:21.723546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.723428 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f6e7109060f166825c9ae8d86c6de4773988050ed542d6775de045daec94c30" Oct 31 22:15:21.723546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.723477 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl" Oct 31 22:15:21.729897 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.729854 4294 scope.go:122] "RemoveContainer" containerID="15952c111596f9b61fd4975b0e619c8ad9eba3926c33277fc1edb11e8d2e827c" Oct 31 22:15:21.745352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.743900 4294 scope.go:122] "RemoveContainer" containerID="d59b7daf04b0a07beafc4e53e116ddbf1b025cac27cb8c9c00e444bf467a8edf" Oct 31 22:15:21.868974 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.868949 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw"] Oct 31 22:15:21.898327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.898281 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd"] Oct 31 22:15:21.911790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.911756 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c744c0-29891415-vb6rd"] Oct 31 22:15:21.919961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.919387 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:21.925911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.925768 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn"] Oct 31 22:15:21.958694 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.958665 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:21.966902 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.966855 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt"] Oct 31 22:15:21.975052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.975022 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b36e62-29891415-g7jdt"] Oct 31 22:15:21.994955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.994916 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" path="/var/lib/kubelet/pods/4312b8c4-a53c-4354-b696-695e71f214d6/volumes" Oct 31 22:15:21.996057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.996031 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa3eb2b-4e44-4796-904f-a9aec7047555-kube-api-access-hqqvq\" (UniqueName: \"kubernetes.io/projected/3aa3eb2b-4e44-4796-904f-a9aec7047555-kube-api-access-hqqvq\") pod \"3aa3eb2b-4e44-4796-904f-a9aec7047555\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " Oct 31 22:15:21.996188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.996152 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aa3eb2b-4e44-4796-904f-a9aec7047555-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3eb2b-4e44-4796-904f-a9aec7047555-maas-api-service-ca\") pod \"3aa3eb2b-4e44-4796-904f-a9aec7047555\" (UID: \"3aa3eb2b-4e44-4796-904f-a9aec7047555\") " Oct 31 22:15:21.996725 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.996699 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aa3eb2b-4e44-4796-904f-a9aec7047555-maas-api-service-ca" pod "3aa3eb2b-4e44-4796-904f-a9aec7047555" (UID: "3aa3eb2b-4e44-4796-904f-a9aec7047555"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:21.997393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.997371 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728fa16a-ae89-4220-8990-7ac122157f07" path="/var/lib/kubelet/pods/728fa16a-ae89-4220-8990-7ac122157f07/volumes" Oct 31 22:15:21.997899 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.997880 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3aa3eb2b-4e44-4796-904f-a9aec7047555-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:21.998012 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.997919 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c519470-3a10-488d-a542-20423d52211a" path="/var/lib/kubelet/pods/7c519470-3a10-488d-a542-20423d52211a/volumes" Oct 31 22:15:21.998443 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:21.998423 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" path="/var/lib/kubelet/pods/d7a1dfb1-a6f3-42fa-8566-bb614078045a/volumes" Oct 31 22:15:22.002352 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.001691 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa3eb2b-4e44-4796-904f-a9aec7047555-kube-api-access-hqqvq" pod "3aa3eb2b-4e44-4796-904f-a9aec7047555" (UID: "3aa3eb2b-4e44-4796-904f-a9aec7047555"). InnerVolumeSpecName "kube-api-access-hqqvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:22.017426 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.017349 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:22.023608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.023544 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp"] Oct 31 22:15:22.029508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.029419 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-4bd51b-29891415-btwfp"] Oct 31 22:15:22.098876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.098841 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845f9969-a52b-4d80-a641-81c2e8020c54-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845f9969-a52b-4d80-a641-81c2e8020c54-maas-api-service-ca\") pod \"845f9969-a52b-4d80-a641-81c2e8020c54\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " Oct 31 22:15:22.099244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.099220 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:22.099378 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.099327 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845f9969-a52b-4d80-a641-81c2e8020c54-maas-api-service-ca" pod "845f9969-a52b-4d80-a641-81c2e8020c54" (UID: "845f9969-a52b-4d80-a641-81c2e8020c54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:22.099457 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.099436 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-kube-api-access-k8rk4\" (UniqueName: \"kubernetes.io/projected/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-kube-api-access-k8rk4\") pod \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " Oct 31 22:15:22.099572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.099556 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845f9969-a52b-4d80-a641-81c2e8020c54-kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/845f9969-a52b-4d80-a641-81c2e8020c54-kube-api-access-z5mms\") pod \"845f9969-a52b-4d80-a641-81c2e8020c54\" (UID: \"845f9969-a52b-4d80-a641-81c2e8020c54\") " Oct 31 22:15:22.099651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.099637 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-maas-api-service-ca\") pod \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\" (UID: \"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1\") " Oct 31 22:15:22.100376 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.100353 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqqvq\" (UniqueName: \"kubernetes.io/projected/3aa3eb2b-4e44-4796-904f-a9aec7047555-kube-api-access-hqqvq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.101390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.101372 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845f9969-a52b-4d80-a641-81c2e8020c54-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.101574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.100860 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-maas-api-service-ca" pod "d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" (UID: "d5a772d8-9d68-4a2f-a241-6f356eb5a2c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:22.103619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.103590 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-kube-api-access-k8rk4" pod "d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" (UID: "d5a772d8-9d68-4a2f-a241-6f356eb5a2c1"). InnerVolumeSpecName "kube-api-access-k8rk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:22.103839 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.103810 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845f9969-a52b-4d80-a641-81c2e8020c54-kube-api-access-z5mms" pod "845f9969-a52b-4d80-a641-81c2e8020c54" (UID: "845f9969-a52b-4d80-a641-81c2e8020c54"). InnerVolumeSpecName "kube-api-access-z5mms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:22.114076 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.114046 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz"] Oct 31 22:15:22.127956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.127922 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c2d1b8-29891415-jpvsz"] Oct 31 22:15:22.195187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.195160 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:22.202288 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.202228 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f0741c6-c011-4c39-b014-0d9034978b03-kube-api-access-vzgfw\" (UniqueName: \"kubernetes.io/projected/9f0741c6-c011-4c39-b014-0d9034978b03-kube-api-access-vzgfw\") pod \"9f0741c6-c011-4c39-b014-0d9034978b03\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " Oct 31 22:15:22.202419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.202400 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f0741c6-c011-4c39-b014-0d9034978b03-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0741c6-c011-4c39-b014-0d9034978b03-maas-api-service-ca\") pod \"9f0741c6-c011-4c39-b014-0d9034978b03\" (UID: \"9f0741c6-c011-4c39-b014-0d9034978b03\") " Oct 31 22:15:22.202896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.202867 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0741c6-c011-4c39-b014-0d9034978b03-maas-api-service-ca" pod "9f0741c6-c011-4c39-b014-0d9034978b03" (UID: "9f0741c6-c011-4c39-b014-0d9034978b03"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:22.203152 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.203133 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f0741c6-c011-4c39-b014-0d9034978b03-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.203223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.203156 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8rk4\" (UniqueName: \"kubernetes.io/projected/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-kube-api-access-k8rk4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.203223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.203171 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mms\" (UniqueName: \"kubernetes.io/projected/845f9969-a52b-4d80-a641-81c2e8020c54-kube-api-access-z5mms\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.203223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.203183 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.208457 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.208431 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0741c6-c011-4c39-b014-0d9034978b03-kube-api-access-vzgfw" pod "9f0741c6-c011-4c39-b014-0d9034978b03" (UID: "9f0741c6-c011-4c39-b014-0d9034978b03"). InnerVolumeSpecName "kube-api-access-vzgfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:22.304704 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.304664 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8db9dfad-1040-4e7c-8203-1497b21954f0-kube-api-access-hn9vx\" (UniqueName: \"kubernetes.io/projected/8db9dfad-1040-4e7c-8203-1497b21954f0-kube-api-access-hn9vx\") pod \"8db9dfad-1040-4e7c-8203-1497b21954f0\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " Oct 31 22:15:22.304865 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.304789 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8db9dfad-1040-4e7c-8203-1497b21954f0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db9dfad-1040-4e7c-8203-1497b21954f0-maas-api-service-ca\") pod \"8db9dfad-1040-4e7c-8203-1497b21954f0\" (UID: \"8db9dfad-1040-4e7c-8203-1497b21954f0\") " Oct 31 22:15:22.305163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.305138 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db9dfad-1040-4e7c-8203-1497b21954f0-maas-api-service-ca" pod "8db9dfad-1040-4e7c-8203-1497b21954f0" (UID: "8db9dfad-1040-4e7c-8203-1497b21954f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:22.305585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.305562 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db9dfad-1040-4e7c-8203-1497b21954f0-maas-api-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.305675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.305591 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzgfw\" (UniqueName: \"kubernetes.io/projected/9f0741c6-c011-4c39-b014-0d9034978b03-kube-api-access-vzgfw\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.306952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.306925 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db9dfad-1040-4e7c-8203-1497b21954f0-kube-api-access-hn9vx" pod "8db9dfad-1040-4e7c-8203-1497b21954f0" (UID: "8db9dfad-1040-4e7c-8203-1497b21954f0"). InnerVolumeSpecName "kube-api-access-hn9vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:22.407957 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.407912 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn9vx\" (UniqueName: \"kubernetes.io/projected/8db9dfad-1040-4e7c-8203-1497b21954f0-kube-api-access-hn9vx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:22.732910 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.732880 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" Oct 31 22:15:22.733406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.732911 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4" event={"ID":"845f9969-a52b-4d80-a641-81c2e8020c54","Type":"ContainerDied","Data":"e7d8d8e2269f1a4e0c04db8a676084764c3ed6b739d8e1e13f275b9691a43f2f"} Oct 31 22:15:22.733406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.732996 4294 scope.go:122] "RemoveContainer" containerID="3e158098f4b00f978812e38d4cd55e7b1bc4a792cc68d307d9f5f81e0bc0c03e" Oct 31 22:15:22.738406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.734292 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" Oct 31 22:15:22.738406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.734293 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw" event={"ID":"9f0741c6-c011-4c39-b014-0d9034978b03","Type":"ContainerDied","Data":"3341cd0f7bbe47fd2db59f0d76fafc915e481172bdeeac8ac9a7efb40b3948bb"} Oct 31 22:15:22.738406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.738180 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" event={"ID":"3aa3eb2b-4e44-4796-904f-a9aec7047555","Type":"ContainerDied","Data":"8b3fe72a6ed98d74b3e0925ba9597dc6d9d5fdc0d6c24ab6024cf928ac6111a9"} Oct 31 22:15:22.738406 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.738272 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv" Oct 31 22:15:22.740923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.740894 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" Oct 31 22:15:22.741024 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.740895 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn" event={"ID":"8db9dfad-1040-4e7c-8203-1497b21954f0","Type":"ContainerDied","Data":"6530ed4dbd2b044fa976ded27fe2202fdd16f8728255092c7ee6674319a0c2c7"} Oct 31 22:15:22.751393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.750938 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" event={"ID":"d5a772d8-9d68-4a2f-a241-6f356eb5a2c1","Type":"ContainerDied","Data":"537efebfe3a05725212fc2f09fe1c9d6dfd3af088ed3f5288413c60564750a5c"} Oct 31 22:15:22.751393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.751028 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g" Oct 31 22:15:22.751393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.751082 4294 scope.go:122] "RemoveContainer" containerID="ebf373eb9f65adbb6c20e2629dd26acab97d76d4466578498e984fd194cbff83" Oct 31 22:15:22.764282 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.764259 4294 scope.go:122] "RemoveContainer" containerID="d63d61b7c33f4d52ff77be5cc545abe0387dcdfd11f57d30e2e30b40c5e918ef" Oct 31 22:15:22.773412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.773385 4294 scope.go:122] "RemoveContainer" containerID="91b3d95c412b84ed26ac173e823e23d17b78ec675864d669ddcd00e1b950c5a7" Oct 31 22:15:22.782673 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.782656 4294 scope.go:122] "RemoveContainer" containerID="0dd9c1ef258ef650b1c30da84143db4cf1b1bd769cb89e80b618e687b41b0f5b" Oct 31 22:15:22.834801 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.834771 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn"] Oct 31 22:15:22.839213 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.839185 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-56a6d4-29891415-5flvn"] Oct 31 22:15:22.872170 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.872129 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv"] Oct 31 22:15:22.878693 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.878669 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-106555-29891415-gfzcv"] Oct 31 22:15:22.911745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.911716 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g"] Oct 31 22:15:22.918084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.918061 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-cc45a7-29891415-m8x6g"] Oct 31 22:15:22.953027 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.952995 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4"] Oct 31 22:15:22.963686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.963661 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f1e791-29891415-kq7p4"] Oct 31 22:15:22.996988 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:22.996913 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw"] Oct 31 22:15:23.001363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.001336 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c16684-29891415-h6hkw"] Oct 31 22:15:23.533888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.533853 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd"] Oct 31 22:15:23.534115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.534091 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" containerID="cri-o://22961a112a97cc771ff397f9944dc2e70f643d9737d85f6f1e4ae4cce404c2e5" gracePeriod=30 Oct 31 22:15:23.987674 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.987644 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" path="/var/lib/kubelet/pods/31db38fe-c0de-4701-b7ed-62b8a7539a1a/volumes" Oct 31 22:15:23.988103 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.988089 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" path="/var/lib/kubelet/pods/3aa3eb2b-4e44-4796-904f-a9aec7047555/volumes" Oct 31 22:15:23.988508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.988495 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" path="/var/lib/kubelet/pods/845f9969-a52b-4d80-a641-81c2e8020c54/volumes" Oct 31 22:15:23.989177 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.989162 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" path="/var/lib/kubelet/pods/8db9dfad-1040-4e7c-8203-1497b21954f0/volumes" Oct 31 22:15:23.989567 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.989554 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" path="/var/lib/kubelet/pods/9f0741c6-c011-4c39-b014-0d9034978b03/volumes" Oct 31 22:15:23.989960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.989947 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" path="/var/lib/kubelet/pods/d5a772d8-9d68-4a2f-a241-6f356eb5a2c1/volumes" Oct 31 22:15:23.990614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:23.990600 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" path="/var/lib/kubelet/pods/f6e5c17f-0945-4321-b1d4-26c06bfd778c/volumes" Oct 31 22:15:24.106085 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106043 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb"] Oct 31 22:15:24.106717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106700 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:24.106717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106716 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106730 4294 cpu_manager.go:414] "containerMap: removing container" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106739 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106747 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106753 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106763 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106768 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106775 4294 cpu_manager.go:414] "containerMap: removing container" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106779 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106789 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106793 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106806 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106813 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106824 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106829 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106837 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106841 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a541b3ad-0cf9-4a2e-b488-f2d54d250c83" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106848 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106852 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106859 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106863 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106869 4294 cpu_manager.go:414] "containerMap: removing container" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106873 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106884 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106890 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106900 4294 cpu_manager.go:414] "containerMap: removing container" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.106898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106905 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106913 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106918 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106924 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106929 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106938 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106942 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106950 4294 cpu_manager.go:414] "containerMap: removing container" podUID="bae7637d-1487-4f9b-acda-cc1a4bd37725" containerName="revoke-keys" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106954 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae7637d-1487-4f9b-acda-cc1a4bd37725" containerName="revoke-keys" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106962 4294 cpu_manager.go:414] "containerMap: removing container" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106966 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106971 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106976 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106986 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.106993 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107001 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107006 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107011 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107015 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107028 4294 cpu_manager.go:414] "containerMap: removing container" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107033 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107038 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107043 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107049 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107055 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107064 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107070 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107077 4294 cpu_manager.go:414] "containerMap: removing container" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107081 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107087 4294 cpu_manager.go:414] "containerMap: removing container" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107091 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107099 4294 cpu_manager.go:414] "containerMap: removing container" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107104 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107110 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107114 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107213 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107225 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107234 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107241 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107247 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107253 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107258 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107263 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107269 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107274 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107279 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107284 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc62938-ef10-4235-91af-d518a045ce95" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107288 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107295 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107314 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae7637d-1487-4f9b-acda-cc1a4bd37725" containerName="revoke-keys" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107322 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107329 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107337 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107344 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107350 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107355 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107361 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107366 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107372 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107378 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107382 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107389 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107394 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="31db38fe-c0de-4701-b7ed-62b8a7539a1a" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107399 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107405 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa3eb2b-4e44-4796-904f-a9aec7047555" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107415 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ee08211-d158-43a4-a055-61d92d2ee655" containerName="cleanup" Oct 31 22:15:24.107759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.107423 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="728fa16a-ae89-4220-8990-7ac122157f07" containerName="cleanup" Oct 31 22:15:24.112695 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.112674 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.129962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.129933 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb"] Oct 31 22:15:24.184633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.184600 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:15:24.184942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.184924 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:15:24.243870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.243788 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxsfd\" (UniqueName: \"kubernetes.io/projected/5471b46c-02ff-44a0-be90-b6caa55a2bdd-kube-api-access-jxsfd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.243870 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.243831 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.244046 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.243997 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.345720 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.345681 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxsfd\" (UniqueName: \"kubernetes.io/projected/5471b46c-02ff-44a0-be90-b6caa55a2bdd-kube-api-access-jxsfd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.345917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.345732 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.345917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.345825 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.346585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.346555 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.348413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.348383 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.369619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.369589 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxsfd\" (UniqueName: \"kubernetes.io/projected/5471b46c-02ff-44a0-be90-b6caa55a2bdd-kube-api-access-jxsfd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.433231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.433196 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:24.737962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.737928 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb"] Oct 31 22:15:24.738172 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:24.738144 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5471b46c_02ff_44a0_be90_b6caa55a2bdd.slice/crio-fd59ada485886fee6881a4cb4f296cdc8af03f65fa6c5fe958b9cf3537d25d33 WatchSource:0}: Error finding container fd59ada485886fee6881a4cb4f296cdc8af03f65fa6c5fe958b9cf3537d25d33: Status 404 returned error can't find the container with id fd59ada485886fee6881a4cb4f296cdc8af03f65fa6c5fe958b9cf3537d25d33 Oct 31 22:15:24.787083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:24.787051 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" event={"ID":"5471b46c-02ff-44a0-be90-b6caa55a2bdd","Type":"ContainerStarted","Data":"fd59ada485886fee6881a4cb4f296cdc8af03f65fa6c5fe958b9cf3537d25d33"} Oct 31 22:15:25.798390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:25.798354 4294 generic.go:356] "Generic (PLEG): container finished" podID="5471b46c-02ff-44a0-be90-b6caa55a2bdd" containerID="cc930a4ceace553996abc7d76161f6b1c0987652e7381f265fd83edea6a11ac2" exitCode=0 Oct 31 22:15:25.798777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:25.798432 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" event={"ID":"5471b46c-02ff-44a0-be90-b6caa55a2bdd","Type":"ContainerDied","Data":"cc930a4ceace553996abc7d76161f6b1c0987652e7381f265fd83edea6a11ac2"} Oct 31 22:15:26.965259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:26.965234 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:27.089231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.089129 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-xks-ca\") pod \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " Oct 31 22:15:27.089231 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.089188 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5471b46c-02ff-44a0-be90-b6caa55a2bdd-kube-api-access-jxsfd\" (UniqueName: \"kubernetes.io/projected/5471b46c-02ff-44a0-be90-b6caa55a2bdd-kube-api-access-jxsfd\") pod \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " Oct 31 22:15:27.089848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.089236 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-openshift-ca\") pod \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\" (UID: \"5471b46c-02ff-44a0-be90-b6caa55a2bdd\") " Oct 31 22:15:27.090419 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.090256 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-openshift-ca" pod "5471b46c-02ff-44a0-be90-b6caa55a2bdd" (UID: "5471b46c-02ff-44a0-be90-b6caa55a2bdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:27.090878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.090845 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:27.096570 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.096539 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-xks-ca" pod "5471b46c-02ff-44a0-be90-b6caa55a2bdd" (UID: "5471b46c-02ff-44a0-be90-b6caa55a2bdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:27.117355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.116815 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5471b46c-02ff-44a0-be90-b6caa55a2bdd-kube-api-access-jxsfd" pod "5471b46c-02ff-44a0-be90-b6caa55a2bdd" (UID: "5471b46c-02ff-44a0-be90-b6caa55a2bdd"). InnerVolumeSpecName "kube-api-access-jxsfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:27.192639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.192608 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5471b46c-02ff-44a0-be90-b6caa55a2bdd-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:27.192639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.192634 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxsfd\" (UniqueName: \"kubernetes.io/projected/5471b46c-02ff-44a0-be90-b6caa55a2bdd-kube-api-access-jxsfd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:27.814787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.814757 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" Oct 31 22:15:27.814787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.814762 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb" event={"ID":"5471b46c-02ff-44a0-be90-b6caa55a2bdd","Type":"ContainerDied","Data":"fd59ada485886fee6881a4cb4f296cdc8af03f65fa6c5fe958b9cf3537d25d33"} Oct 31 22:15:27.815056 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.814824 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd59ada485886fee6881a4cb4f296cdc8af03f65fa6c5fe958b9cf3537d25d33" Oct 31 22:15:27.827093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:27.827066 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt"] Oct 31 22:15:28.825218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.825189 4294 generic.go:356] "Generic (PLEG): container finished" podID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerID="22961a112a97cc771ff397f9944dc2e70f643d9737d85f6f1e4ae4cce404c2e5" exitCode=0 Oct 31 22:15:28.825687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.825261 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" event={"ID":"a59a4817-e90a-4639-8abe-7f1584fa5eb6","Type":"ContainerDied","Data":"22961a112a97cc771ff397f9944dc2e70f643d9737d85f6f1e4ae4cce404c2e5"} Oct 31 22:15:28.825687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.825338 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" event={"ID":"a59a4817-e90a-4639-8abe-7f1584fa5eb6","Type":"ContainerDied","Data":"ac4627d72b9d1e5a97a400e273b7d3197b9aef1f477dcddf6aecbb48b30a4c83"} Oct 31 22:15:28.825687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.825360 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac4627d72b9d1e5a97a400e273b7d3197b9aef1f477dcddf6aecbb48b30a4c83" Oct 31 22:15:28.825687 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.825376 4294 scope.go:122] "RemoveContainer" containerID="95d235eab51805239e7b995f5eeb1c7692cbc39ebdd6ba2089788d2411b31379" Oct 31 22:15:28.827347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.827326 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939361 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-credential-socket\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939451 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-kube-api-access-dnl5s\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-kube-api-access-dnl5s\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939540 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istiod-ca-cert\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939587 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-token\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939747 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-envoy\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939815 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-certs\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939843 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-podinfo\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939873 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-data\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.939910 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-socket\") pod \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\" (UID: \"a59a4817-e90a-4639-8abe-7f1584fa5eb6\") " Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.948742 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-certs" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.950916 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-envoy" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.951225 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istiod-ca-cert" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.951278 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-credential-socket" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.951339 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-data" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:28.953189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.952555 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-podinfo" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:15:28.954138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.953995 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-socket" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:28.954196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.954176 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-kube-api-access-dnl5s" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "kube-api-access-dnl5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:28.954474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:28.954424 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-token" pod "a59a4817-e90a-4639-8abe-7f1584fa5eb6" (UID: "a59a4817-e90a-4639-8abe-7f1584fa5eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:29.042102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042073 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042101 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042112 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042123 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042135 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042146 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042158 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042181 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a59a4817-e90a-4639-8abe-7f1584fa5eb6-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.042328 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.042204 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnl5s\" (UniqueName: \"kubernetes.io/projected/a59a4817-e90a-4639-8abe-7f1584fa5eb6-kube-api-access-dnl5s\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:29.834587 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.834555 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd" Oct 31 22:15:29.930344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.930296 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd"] Oct 31 22:15:29.935048 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.935021 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a5affe-openshift-default-dd9b47fd9-6q2pd"] Oct 31 22:15:29.985895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:29.985857 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" path="/var/lib/kubelet/pods/a59a4817-e90a-4639-8abe-7f1584fa5eb6/volumes" Oct 31 22:15:31.311191 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:31.311153 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:15:32.827848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:32.827777 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" podUID="9f451d27-023c-467d-b582-d8373e0da7fe" containerName="payload-processing" containerID="cri-o://e0af8a652c62fd737a999a3bb8b41748f9360e2895fa12fc044eb99e3391c241" gracePeriod=25 Oct 31 22:15:34.080439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:34.080403 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" podUID="9f451d27-023c-467d-b582-d8373e0da7fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:15:43.423583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:43.423550 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl"] Oct 31 22:15:43.435097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:43.435070 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-710927-q5rsl"] Oct 31 22:15:43.985806 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:43.985776 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae7637d-1487-4f9b-acda-cc1a4bd37725" path="/var/lib/kubelet/pods/bae7637d-1487-4f9b-acda-cc1a4bd37725/volumes" Oct 31 22:15:44.030490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:44.030462 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt"] Oct 31 22:15:44.042065 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:44.042035 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a60dc9-lsvwt"] Oct 31 22:15:44.080372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:44.080341 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" podUID="9f451d27-023c-467d-b582-d8373e0da7fe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:15:45.985358 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:45.985292 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b50a00-9932-429c-933f-97752e1e4923" path="/var/lib/kubelet/pods/03b50a00-9932-429c-933f-97752e1e4923/volumes" Oct 31 22:15:51.190339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.190275 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz"] Oct 31 22:15:51.191511 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.191468 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" containerID="cri-o://9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a" gracePeriod=30 Oct 31 22:15:51.760023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.759988 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7"] Oct 31 22:15:51.760582 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760568 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5471b46c-02ff-44a0-be90-b6caa55a2bdd" containerName="revoke-keys" Oct 31 22:15:51.760635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760585 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5471b46c-02ff-44a0-be90-b6caa55a2bdd" containerName="revoke-keys" Oct 31 22:15:51.760635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760598 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:51.760635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760603 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f0741c6-c011-4c39-b014-0d9034978b03" containerName="cleanup" Oct 31 22:15:51.760635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760611 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:51.760635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760617 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:51.760635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760626 4294 cpu_manager.go:414] "containerMap: removing container" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:51.760635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760631 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760639 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760644 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760653 4294 cpu_manager.go:414] "containerMap: removing container" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760657 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760665 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760670 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760680 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760687 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4312b8c4-a53c-4354-b696-695e71f214d6" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760705 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760709 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760815 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c519470-3a10-488d-a542-20423d52211a" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760828 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a1dfb1-a6f3-42fa-8566-bb614078045a" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760837 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760845 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a772d8-9d68-4a2f-a241-6f356eb5a2c1" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760853 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8db9dfad-1040-4e7c-8203-1497b21954f0" containerName="cleanup" Oct 31 22:15:51.760855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760860 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5471b46c-02ff-44a0-be90-b6caa55a2bdd" containerName="revoke-keys" Oct 31 22:15:51.761413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760871 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e5c17f-0945-4321-b1d4-26c06bfd778c" containerName="cleanup" Oct 31 22:15:51.761413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.760880 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="845f9969-a52b-4d80-a641-81c2e8020c54" containerName="cleanup" Oct 31 22:15:51.765592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.765573 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.783192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.783165 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7"] Oct 31 22:15:51.827634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.827608 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:15:51.827748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.827735 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:15:51.863383 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.863358 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.863483 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.863392 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68t9\" (UniqueName: \"kubernetes.io/projected/133120f7-6696-4b70-920a-983848e2a1a2-kube-api-access-h68t9\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.863559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.863537 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.964721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.964697 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.964836 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.964732 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68t9\" (UniqueName: \"kubernetes.io/projected/133120f7-6696-4b70-920a-983848e2a1a2-kube-api-access-h68t9\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.964836 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.964800 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.965457 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.965437 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.966938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.966910 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:51.988202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:51.988180 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68t9\" (UniqueName: \"kubernetes.io/projected/133120f7-6696-4b70-920a-983848e2a1a2-kube-api-access-h68t9\") pod \"maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:52.083508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:52.083438 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:52.398321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:52.398283 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7"] Oct 31 22:15:52.399550 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:15:52.399524 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133120f7_6696_4b70_920a_983848e2a1a2.slice/crio-2049189181a7cd84584869b47950ef010b5cc77cbe0a2342d55c14251cbd5b54 WatchSource:0}: Error finding container 2049189181a7cd84584869b47950ef010b5cc77cbe0a2342d55c14251cbd5b54: Status 404 returned error can't find the container with id 2049189181a7cd84584869b47950ef010b5cc77cbe0a2342d55c14251cbd5b54 Oct 31 22:15:53.042648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.042616 4294 generic.go:356] "Generic (PLEG): container finished" podID="9f451d27-023c-467d-b582-d8373e0da7fe" containerID="e0af8a652c62fd737a999a3bb8b41748f9360e2895fa12fc044eb99e3391c241" exitCode=0 Oct 31 22:15:53.042842 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.042685 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" event={"ID":"9f451d27-023c-467d-b582-d8373e0da7fe","Type":"ContainerDied","Data":"e0af8a652c62fd737a999a3bb8b41748f9360e2895fa12fc044eb99e3391c241"} Oct 31 22:15:53.044778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.044750 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" event={"ID":"133120f7-6696-4b70-920a-983848e2a1a2","Type":"ContainerStarted","Data":"072b88d58851405dbf7340fc4d288fddbc62134511d93cb82aaeb2472f81a2cd"} Oct 31 22:15:53.044931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.044798 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" event={"ID":"133120f7-6696-4b70-920a-983848e2a1a2","Type":"ContainerStarted","Data":"2049189181a7cd84584869b47950ef010b5cc77cbe0a2342d55c14251cbd5b54"} Oct 31 22:15:53.086982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.086939 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" podStartSLOduration=2.086926436 podStartE2EDuration="2.086926436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:15:53.085537414 +0000 UTC m=+3593.210644669" watchObservedRunningTime="2026-10-31 22:15:53.086926436 +0000 UTC m=+3593.212033667" Oct 31 22:15:53.097368 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.097346 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:15:53.181097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.181050 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f451d27-023c-467d-b582-d8373e0da7fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f451d27-023c-467d-b582-d8373e0da7fe-plugins-config-volume\") pod \"9f451d27-023c-467d-b582-d8373e0da7fe\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " Oct 31 22:15:53.182964 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.181136 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f451d27-023c-467d-b582-d8373e0da7fe-kube-api-access-m4r2f\" (UniqueName: \"kubernetes.io/projected/9f451d27-023c-467d-b582-d8373e0da7fe-kube-api-access-m4r2f\") pod \"9f451d27-023c-467d-b582-d8373e0da7fe\" (UID: \"9f451d27-023c-467d-b582-d8373e0da7fe\") " Oct 31 22:15:53.183675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.183643 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f451d27-023c-467d-b582-d8373e0da7fe-plugins-config-volume" pod "9f451d27-023c-467d-b582-d8373e0da7fe" (UID: "9f451d27-023c-467d-b582-d8373e0da7fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:53.186452 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.186428 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f451d27-023c-467d-b582-d8373e0da7fe-kube-api-access-m4r2f" pod "9f451d27-023c-467d-b582-d8373e0da7fe" (UID: "9f451d27-023c-467d-b582-d8373e0da7fe"). InnerVolumeSpecName "kube-api-access-m4r2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:53.283594 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.283564 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f451d27-023c-467d-b582-d8373e0da7fe-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:53.283594 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.283592 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4r2f\" (UniqueName: \"kubernetes.io/projected/9f451d27-023c-467d-b582-d8373e0da7fe-kube-api-access-m4r2f\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:53.951792 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.951754 4294 patch_prober.go:29] interesting pod/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:15:53.952173 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:53.951812 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:15:54.053881 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:54.053853 4294 generic.go:356] "Generic (PLEG): container finished" podID="133120f7-6696-4b70-920a-983848e2a1a2" containerID="072b88d58851405dbf7340fc4d288fddbc62134511d93cb82aaeb2472f81a2cd" exitCode=0 Oct 31 22:15:54.054024 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:54.053936 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" event={"ID":"133120f7-6696-4b70-920a-983848e2a1a2","Type":"ContainerDied","Data":"072b88d58851405dbf7340fc4d288fddbc62134511d93cb82aaeb2472f81a2cd"} Oct 31 22:15:54.055395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:54.055378 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" Oct 31 22:15:54.055508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:54.055423 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt" event={"ID":"9f451d27-023c-467d-b582-d8373e0da7fe","Type":"ContainerDied","Data":"43ba612ba40d6a9834d34dfed58f7a13792394955a84ac1b00a8f0ba7ffec792"} Oct 31 22:15:54.055508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:54.055482 4294 scope.go:122] "RemoveContainer" containerID="e0af8a652c62fd737a999a3bb8b41748f9360e2895fa12fc044eb99e3391c241" Oct 31 22:15:54.131798 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:54.131766 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt"] Oct 31 22:15:54.135603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:54.135576 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-710927-7859f7d845-ht9pt"] Oct 31 22:15:55.031755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.031724 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt"] Oct 31 22:15:55.043767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.043739 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5be66063-qwmxt"] Oct 31 22:15:55.215703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.215679 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:55.320773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.320701 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133120f7-6696-4b70-920a-983848e2a1a2-kube-api-access-h68t9\" (UniqueName: \"kubernetes.io/projected/133120f7-6696-4b70-920a-983848e2a1a2-kube-api-access-h68t9\") pod \"133120f7-6696-4b70-920a-983848e2a1a2\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " Oct 31 22:15:55.320913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.320894 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-xks-ca\") pod \"133120f7-6696-4b70-920a-983848e2a1a2\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " Oct 31 22:15:55.321057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.321046 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-openshift-ca\") pod \"133120f7-6696-4b70-920a-983848e2a1a2\" (UID: \"133120f7-6696-4b70-920a-983848e2a1a2\") " Oct 31 22:15:55.321416 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.321390 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-openshift-ca" pod "133120f7-6696-4b70-920a-983848e2a1a2" (UID: "133120f7-6696-4b70-920a-983848e2a1a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:55.321660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.321638 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:55.322911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.322884 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-xks-ca" pod "133120f7-6696-4b70-920a-983848e2a1a2" (UID: "133120f7-6696-4b70-920a-983848e2a1a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:15:55.322911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.322899 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133120f7-6696-4b70-920a-983848e2a1a2-kube-api-access-h68t9" pod "133120f7-6696-4b70-920a-983848e2a1a2" (UID: "133120f7-6696-4b70-920a-983848e2a1a2"). InnerVolumeSpecName "kube-api-access-h68t9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:55.423187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.423158 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/133120f7-6696-4b70-920a-983848e2a1a2-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:55.423187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.423184 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68t9\" (UniqueName: \"kubernetes.io/projected/133120f7-6696-4b70-920a-983848e2a1a2-kube-api-access-h68t9\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:55.986842 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.986808 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f451d27-023c-467d-b582-d8373e0da7fe" path="/var/lib/kubelet/pods/9f451d27-023c-467d-b582-d8373e0da7fe/volumes" Oct 31 22:15:55.987284 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:55.987269 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f47986-a737-458f-92a2-dcfa08165bc1" path="/var/lib/kubelet/pods/a0f47986-a737-458f-92a2-dcfa08165bc1/volumes" Oct 31 22:15:56.073580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.073557 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" Oct 31 22:15:56.073892 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.073600 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7" event={"ID":"133120f7-6696-4b70-920a-983848e2a1a2","Type":"ContainerDied","Data":"2049189181a7cd84584869b47950ef010b5cc77cbe0a2342d55c14251cbd5b54"} Oct 31 22:15:56.073892 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.073642 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2049189181a7cd84584869b47950ef010b5cc77cbe0a2342d55c14251cbd5b54" Oct 31 22:15:56.468420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.468395 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:15:56.637356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637243 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-podinfo\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637286 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-envoy\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637387 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istiod-ca-cert\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637436 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-kube-api-access-w9fd7\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-kube-api-access-w9fd7\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637473 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-certs\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637499 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-data\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637636 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-token\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-token\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637731 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-socket\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.637848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637803 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-credential-socket\") pod \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\" (UID: \"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e\") " Oct 31 22:15:56.638005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637848 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-certs" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:56.638005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637868 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-data" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:56.638005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.637855 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istiod-ca-cert" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:15:56.638259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.638181 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-socket" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:56.638394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.638272 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-credential-socket" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:56.638394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.638360 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.638394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.638382 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.638532 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.638394 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.638532 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.638408 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.639620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.639597 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-podinfo" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:15:56.639723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.639636 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-envoy" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:15:56.640004 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.639986 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-kube-api-access-w9fd7" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "kube-api-access-w9fd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:56.640138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.640118 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-token" pod "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" (UID: "eed8ab0c-ab5d-4109-9938-db44d2ca0e6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:15:56.739445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.739424 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.739445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.739445 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.739566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.739454 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.739566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.739463 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:56.739566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:56.739472 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9fd7\" (UniqueName: \"kubernetes.io/projected/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e-kube-api-access-w9fd7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:15:57.081407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.081376 4294 generic.go:356] "Generic (PLEG): container finished" podID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerID="9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a" exitCode=0 Oct 31 22:15:57.081869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.081415 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" Oct 31 22:15:57.081869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.081423 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" event={"ID":"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e","Type":"ContainerDied","Data":"9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a"} Oct 31 22:15:57.081869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.081474 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz" event={"ID":"eed8ab0c-ab5d-4109-9938-db44d2ca0e6e","Type":"ContainerDied","Data":"489cff26bd4b318f726f0b94e4ca0d430c13740efa6d50d85f7d2f761aacc49c"} Oct 31 22:15:57.081869 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.081507 4294 scope.go:122] "RemoveContainer" containerID="9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a" Oct 31 22:15:57.092882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.092864 4294 scope.go:122] "RemoveContainer" containerID="9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a" Oct 31 22:15:57.093158 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:15:57.093139 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a\": container with ID starting with 9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a not found: ID does not exist" containerID="9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a" Oct 31 22:15:57.093220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.093164 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a"} err="failed to get container status \"9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a\": rpc error: code = NotFound desc = could not find container \"9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a\": container with ID starting with 9593eea320091d2890f3e0577bc445c200c5ecf07a8a47ff40887e208ddca01a not found: ID does not exist" Oct 31 22:15:57.146955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.146928 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz"] Oct 31 22:15:57.152778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.152036 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-710927-openshift-default-6b78cf7667-bfthz"] Oct 31 22:15:57.985739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:15:57.985711 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" path="/var/lib/kubelet/pods/eed8ab0c-ab5d-4109-9938-db44d2ca0e6e/volumes" Oct 31 22:16:00.646828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:00.646804 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:16:00.688703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:00.688677 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:16:01.310237 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:16:01.310201 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:16:02.230290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:02.230252 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx"] Oct 31 22:16:02.265221 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:02.265193 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp"] Oct 31 22:16:03.753997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:03.753965 4294 scope.go:122] "RemoveContainer" containerID="a4f4c23b0daf71e2d659b9e5306aaeb52360af943723389c7d41192fee09c54d" Oct 31 22:16:03.767198 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:03.767172 4294 scope.go:122] "RemoveContainer" containerID="22961a112a97cc771ff397f9944dc2e70f643d9737d85f6f1e4ae4cce404c2e5" Oct 31 22:16:03.777327 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:03.777287 4294 scope.go:122] "RemoveContainer" containerID="b907e1a8cefdf155866f8f7bead2cb33ceda57d6b90eac115b9e428f85f91687" Oct 31 22:16:03.786079 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:03.786059 4294 scope.go:122] "RemoveContainer" containerID="f402969b4106ae47c5b0c5328daae159cb5d04485aefc98781b47dfcd0fb0cf1" Oct 31 22:16:05.540264 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:05.540232 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb"] Oct 31 22:16:05.551684 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:05.551661 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-6b5dab-qrtbb"] Oct 31 22:16:05.986775 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:05.986747 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5471b46c-02ff-44a0-be90-b6caa55a2bdd" path="/var/lib/kubelet/pods/5471b46c-02ff-44a0-be90-b6caa55a2bdd/volumes" Oct 31 22:16:07.230970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:07.230916 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" podUID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" containerName="payload-processing" containerID="cri-o://17af1cf45fe28dd9848cfa23d1684fde296fd207e10485e5ae52c709d54deabc" gracePeriod=25 Oct 31 22:16:07.265709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:07.265674 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" podUID="62125f24-856d-4185-86c0-3e9f46bcb751" containerName="payload-pre-processing" containerID="cri-o://6778008494f716a0395f9c4c909e026ce6458f54decfabc3790df35986487292" gracePeriod=25 Oct 31 22:16:13.451626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.451595 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc"] Oct 31 22:16:13.452160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452145 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" Oct 31 22:16:13.452216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452160 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" Oct 31 22:16:13.452216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452175 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9f451d27-023c-467d-b582-d8373e0da7fe" containerName="payload-processing" Oct 31 22:16:13.452216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452179 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f451d27-023c-467d-b582-d8373e0da7fe" containerName="payload-processing" Oct 31 22:16:13.452216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452202 4294 cpu_manager.go:414] "containerMap: removing container" podUID="133120f7-6696-4b70-920a-983848e2a1a2" containerName="revoke-keys" Oct 31 22:16:13.452216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452207 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133120f7-6696-4b70-920a-983848e2a1a2" containerName="revoke-keys" Oct 31 22:16:13.452412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452219 4294 cpu_manager.go:414] "containerMap: removing container" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" Oct 31 22:16:13.452412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452225 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" Oct 31 22:16:13.452412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452321 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="133120f7-6696-4b70-920a-983848e2a1a2" containerName="revoke-keys" Oct 31 22:16:13.452412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452329 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59a4817-e90a-4639-8abe-7f1584fa5eb6" containerName="istio-proxy" Oct 31 22:16:13.452412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452337 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed8ab0c-ab5d-4109-9938-db44d2ca0e6e" containerName="istio-proxy" Oct 31 22:16:13.452412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.452349 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f451d27-023c-467d-b582-d8373e0da7fe" containerName="payload-processing" Oct 31 22:16:13.455755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.455740 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.474402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.474381 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc"] Oct 31 22:16:13.527628 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.527574 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:16:13.527823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.527807 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:16:13.559956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.559927 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.560088 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.560066 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2h5\" (UniqueName: \"kubernetes.io/projected/e114a5c1-a188-4f54-b365-9c5e4828431f-kube-api-access-4v2h5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.560341 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.560286 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.661710 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.661683 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2h5\" (UniqueName: \"kubernetes.io/projected/e114a5c1-a188-4f54-b365-9c5e4828431f-kube-api-access-4v2h5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.661835 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.661822 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.661879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.661854 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.662566 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.662535 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.664287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.664258 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.684609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.684581 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2h5\" (UniqueName: \"kubernetes.io/projected/e114a5c1-a188-4f54-b365-9c5e4828431f-kube-api-access-4v2h5\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:13.769324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:13.769277 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:14.073496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:14.071359 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc"] Oct 31 22:16:14.076792 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:16:14.076767 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode114a5c1_a188_4f54_b365_9c5e4828431f.slice/crio-9ddbc0bb7445271c9a4a02e7e6a2233c1adda8ba5a4cebe5ee3ba1be809074a4 WatchSource:0}: Error finding container 9ddbc0bb7445271c9a4a02e7e6a2233c1adda8ba5a4cebe5ee3ba1be809074a4: Status 404 returned error can't find the container with id 9ddbc0bb7445271c9a4a02e7e6a2233c1adda8ba5a4cebe5ee3ba1be809074a4 Oct 31 22:16:14.287810 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:14.287777 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" event={"ID":"e114a5c1-a188-4f54-b365-9c5e4828431f","Type":"ContainerStarted","Data":"9ddbc0bb7445271c9a4a02e7e6a2233c1adda8ba5a4cebe5ee3ba1be809074a4"} Oct 31 22:16:15.296613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:15.296577 4294 generic.go:356] "Generic (PLEG): container finished" podID="e114a5c1-a188-4f54-b365-9c5e4828431f" containerID="db3f6c34ce4521a4bf7b44e71468715de0216f827baef862eeb54f903435f9de" exitCode=0 Oct 31 22:16:15.296997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:15.296660 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" event={"ID":"e114a5c1-a188-4f54-b365-9c5e4828431f","Type":"ContainerDied","Data":"db3f6c34ce4521a4bf7b44e71468715de0216f827baef862eeb54f903435f9de"} Oct 31 22:16:16.369639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.369331 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" podUID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:16.370677 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.370644 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" podUID="62125f24-856d-4185-86c0-3e9f46bcb751" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:16.482122 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.482065 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:16.598873 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.598792 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-xks-ca\") pod \"e114a5c1-a188-4f54-b365-9c5e4828431f\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " Oct 31 22:16:16.599044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.598909 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e114a5c1-a188-4f54-b365-9c5e4828431f-kube-api-access-4v2h5\" (UniqueName: \"kubernetes.io/projected/e114a5c1-a188-4f54-b365-9c5e4828431f-kube-api-access-4v2h5\") pod \"e114a5c1-a188-4f54-b365-9c5e4828431f\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " Oct 31 22:16:16.599044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.598971 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-openshift-ca\") pod \"e114a5c1-a188-4f54-b365-9c5e4828431f\" (UID: \"e114a5c1-a188-4f54-b365-9c5e4828431f\") " Oct 31 22:16:16.599394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.599359 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-openshift-ca" pod "e114a5c1-a188-4f54-b365-9c5e4828431f" (UID: "e114a5c1-a188-4f54-b365-9c5e4828431f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:16.600945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.600910 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-xks-ca" pod "e114a5c1-a188-4f54-b365-9c5e4828431f" (UID: "e114a5c1-a188-4f54-b365-9c5e4828431f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:16.601029 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.600989 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e114a5c1-a188-4f54-b365-9c5e4828431f-kube-api-access-4v2h5" pod "e114a5c1-a188-4f54-b365-9c5e4828431f" (UID: "e114a5c1-a188-4f54-b365-9c5e4828431f"). InnerVolumeSpecName "kube-api-access-4v2h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:16.700446 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.700418 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:16.700446 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.700447 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v2h5\" (UniqueName: \"kubernetes.io/projected/e114a5c1-a188-4f54-b365-9c5e4828431f-kube-api-access-4v2h5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:16.700591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:16.700459 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e114a5c1-a188-4f54-b365-9c5e4828431f-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:17.311921 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:17.311889 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" Oct 31 22:16:17.311921 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:17.311895 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc" event={"ID":"e114a5c1-a188-4f54-b365-9c5e4828431f","Type":"ContainerDied","Data":"9ddbc0bb7445271c9a4a02e7e6a2233c1adda8ba5a4cebe5ee3ba1be809074a4"} Oct 31 22:16:17.312136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:17.311945 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ddbc0bb7445271c9a4a02e7e6a2233c1adda8ba5a4cebe5ee3ba1be809074a4" Oct 31 22:16:23.916749 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:23.916711 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b"] Oct 31 22:16:26.369131 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:26.369094 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" podUID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:26.369631 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:26.369263 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" podUID="62125f24-856d-4185-86c0-3e9f46bcb751" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:27.475755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.475534 4294 generic.go:356] "Generic (PLEG): container finished" podID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" containerID="17af1cf45fe28dd9848cfa23d1684fde296fd207e10485e5ae52c709d54deabc" exitCode=0 Oct 31 22:16:27.475755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.475678 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" event={"ID":"af26ed87-0e43-40ef-9c75-3f6acc1464a4","Type":"ContainerDied","Data":"17af1cf45fe28dd9848cfa23d1684fde296fd207e10485e5ae52c709d54deabc"} Oct 31 22:16:27.478580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.478345 4294 generic.go:356] "Generic (PLEG): container finished" podID="62125f24-856d-4185-86c0-3e9f46bcb751" containerID="6778008494f716a0395f9c4c909e026ce6458f54decfabc3790df35986487292" exitCode=0 Oct 31 22:16:27.478580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.478453 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" event={"ID":"62125f24-856d-4185-86c0-3e9f46bcb751","Type":"ContainerDied","Data":"6778008494f716a0395f9c4c909e026ce6458f54decfabc3790df35986487292"} Oct 31 22:16:27.579978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.579946 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:16:27.630136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.630055 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:16:27.674932 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.674897 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af26ed87-0e43-40ef-9c75-3f6acc1464a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af26ed87-0e43-40ef-9c75-3f6acc1464a4-plugins-config-volume\") pod \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " Oct 31 22:16:27.675122 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.675040 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af26ed87-0e43-40ef-9c75-3f6acc1464a4-kube-api-access-x7rgk\" (UniqueName: \"kubernetes.io/projected/af26ed87-0e43-40ef-9c75-3f6acc1464a4-kube-api-access-x7rgk\") pod \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\" (UID: \"af26ed87-0e43-40ef-9c75-3f6acc1464a4\") " Oct 31 22:16:27.675403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.675275 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af26ed87-0e43-40ef-9c75-3f6acc1464a4-plugins-config-volume" pod "af26ed87-0e43-40ef-9c75-3f6acc1464a4" (UID: "af26ed87-0e43-40ef-9c75-3f6acc1464a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:27.675778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.675759 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af26ed87-0e43-40ef-9c75-3f6acc1464a4-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:27.677157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.677128 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af26ed87-0e43-40ef-9c75-3f6acc1464a4-kube-api-access-x7rgk" pod "af26ed87-0e43-40ef-9c75-3f6acc1464a4" (UID: "af26ed87-0e43-40ef-9c75-3f6acc1464a4"). InnerVolumeSpecName "kube-api-access-x7rgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:27.777226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.777124 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62125f24-856d-4185-86c0-3e9f46bcb751-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62125f24-856d-4185-86c0-3e9f46bcb751-plugins-config-volume\") pod \"62125f24-856d-4185-86c0-3e9f46bcb751\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " Oct 31 22:16:27.777226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.777234 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62125f24-856d-4185-86c0-3e9f46bcb751-kube-api-access-m5ddw\" (UniqueName: \"kubernetes.io/projected/62125f24-856d-4185-86c0-3e9f46bcb751-kube-api-access-m5ddw\") pod \"62125f24-856d-4185-86c0-3e9f46bcb751\" (UID: \"62125f24-856d-4185-86c0-3e9f46bcb751\") " Oct 31 22:16:27.777585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.777559 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62125f24-856d-4185-86c0-3e9f46bcb751-plugins-config-volume" pod "62125f24-856d-4185-86c0-3e9f46bcb751" (UID: "62125f24-856d-4185-86c0-3e9f46bcb751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:27.777922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.777904 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62125f24-856d-4185-86c0-3e9f46bcb751-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:27.778003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.777926 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7rgk\" (UniqueName: \"kubernetes.io/projected/af26ed87-0e43-40ef-9c75-3f6acc1464a4-kube-api-access-x7rgk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:27.779317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.779278 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62125f24-856d-4185-86c0-3e9f46bcb751-kube-api-access-m5ddw" pod "62125f24-856d-4185-86c0-3e9f46bcb751" (UID: "62125f24-856d-4185-86c0-3e9f46bcb751"). InnerVolumeSpecName "kube-api-access-m5ddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:27.879222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:27.879187 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ddw\" (UniqueName: \"kubernetes.io/projected/62125f24-856d-4185-86c0-3e9f46bcb751-kube-api-access-m5ddw\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:28.491183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.491152 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" Oct 31 22:16:28.491183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.491175 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx" event={"ID":"af26ed87-0e43-40ef-9c75-3f6acc1464a4","Type":"ContainerDied","Data":"30481aced7f48ad1ea79f3fada21c81c98f9c579df7ae29a5d721c9c111b1a8a"} Oct 31 22:16:28.491781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.491246 4294 scope.go:122] "RemoveContainer" containerID="17af1cf45fe28dd9848cfa23d1684fde296fd207e10485e5ae52c709d54deabc" Oct 31 22:16:28.493369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.493345 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" event={"ID":"62125f24-856d-4185-86c0-3e9f46bcb751","Type":"ContainerDied","Data":"c7bf1a91fd3fb3ba714fd4dca20f0c6c1de9500b71f0ee48b7cec39d47d41cd2"} Oct 31 22:16:28.493497 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.493403 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp" Oct 31 22:16:28.503168 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.503147 4294 scope.go:122] "RemoveContainer" containerID="6778008494f716a0395f9c4c909e026ce6458f54decfabc3790df35986487292" Oct 31 22:16:28.543332 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.543273 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx"] Oct 31 22:16:28.555528 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.555501 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c2d1b8-79c64d499f-fd4zx"] Oct 31 22:16:28.582611 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.582583 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp"] Oct 31 22:16:28.586413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.586388 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c2d1b8-855f9987b7-s9xgp"] Oct 31 22:16:28.794369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.793945 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf"] Oct 31 22:16:28.794369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.794194 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" containerID="cri-o://9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d" gracePeriod=30 Oct 31 22:16:28.917767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:28.917706 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" podUID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" containerName="payload-processing" containerID="cri-o://3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764" gracePeriod=25 Oct 31 22:16:29.242788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:29.242752 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7"] Oct 31 22:16:29.245861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:29.245839 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c2d1b8-d5xb7"] Oct 31 22:16:29.986349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:29.986294 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133120f7-6696-4b70-920a-983848e2a1a2" path="/var/lib/kubelet/pods/133120f7-6696-4b70-920a-983848e2a1a2/volumes" Oct 31 22:16:29.986781 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:29.986765 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62125f24-856d-4185-86c0-3e9f46bcb751" path="/var/lib/kubelet/pods/62125f24-856d-4185-86c0-3e9f46bcb751/volumes" Oct 31 22:16:29.987155 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:29.987141 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" path="/var/lib/kubelet/pods/af26ed87-0e43-40ef-9c75-3f6acc1464a4/volumes" Oct 31 22:16:31.311061 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:16:31.311031 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:16:31.594002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:31.593927 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" podUID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:33.269034 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.268998 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v"] Oct 31 22:16:33.269598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269585 4294 cpu_manager.go:414] "containerMap: removing container" podUID="62125f24-856d-4185-86c0-3e9f46bcb751" containerName="payload-pre-processing" Oct 31 22:16:33.269649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269599 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62125f24-856d-4185-86c0-3e9f46bcb751" containerName="payload-pre-processing" Oct 31 22:16:33.269649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269630 4294 cpu_manager.go:414] "containerMap: removing container" podUID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" containerName="payload-processing" Oct 31 22:16:33.269649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269635 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" containerName="payload-processing" Oct 31 22:16:33.269649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269643 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e114a5c1-a188-4f54-b365-9c5e4828431f" containerName="revoke-keys" Oct 31 22:16:33.269649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269648 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e114a5c1-a188-4f54-b365-9c5e4828431f" containerName="revoke-keys" Oct 31 22:16:33.269825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269725 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="af26ed87-0e43-40ef-9c75-3f6acc1464a4" containerName="payload-processing" Oct 31 22:16:33.269825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269733 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e114a5c1-a188-4f54-b365-9c5e4828431f" containerName="revoke-keys" Oct 31 22:16:33.269825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.269743 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="62125f24-856d-4185-86c0-3e9f46bcb751" containerName="payload-pre-processing" Oct 31 22:16:33.275574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.275554 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.290724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.290695 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v"] Oct 31 22:16:33.298985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.298960 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:16:33.299131 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.299084 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:16:33.361651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.361624 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.361833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.361798 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kszr\" (UniqueName: \"kubernetes.io/projected/71ce4284-2fc4-45e8-954e-596c3d17d73d-kube-api-access-8kszr\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.361833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.361818 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.462902 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.462874 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.463032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.462998 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kszr\" (UniqueName: \"kubernetes.io/projected/71ce4284-2fc4-45e8-954e-596c3d17d73d-kube-api-access-8kszr\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.463032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.463016 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.463620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.463598 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.465065 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.465045 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.486357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.486335 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kszr\" (UniqueName: \"kubernetes.io/projected/71ce4284-2fc4-45e8-954e-596c3d17d73d-kube-api-access-8kszr\") pod \"maas-api-revoke-sub-7084ad42-jcg6v\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.588789 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.588732 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:33.896064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:33.896036 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v"] Oct 31 22:16:33.897414 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:16:33.897389 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ce4284_2fc4_45e8_954e_596c3d17d73d.slice/crio-b23ed35c2da25c908871c2567f2e0c0dffa6143eeb8d34da6f88e304d2310d32 WatchSource:0}: Error finding container b23ed35c2da25c908871c2567f2e0c0dffa6143eeb8d34da6f88e304d2310d32: Status 404 returned error can't find the container with id b23ed35c2da25c908871c2567f2e0c0dffa6143eeb8d34da6f88e304d2310d32 Oct 31 22:16:34.056423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.056398 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:16:34.190356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190319 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-data\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190383 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-envoy\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190514 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b584485-3ab6-4857-8701-7ec674cabe77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b584485-3ab6-4857-8701-7ec674cabe77-istio-podinfo\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190546 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-kube-api-access-5plww\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-kube-api-access-5plww\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190744 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190577 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-data" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.190744 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190646 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b584485-3ab6-4857-8701-7ec674cabe77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b584485-3ab6-4857-8701-7ec674cabe77-istiod-ca-cert\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190744 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190710 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-istio-token\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-istio-token\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190748 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-certs\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190789 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-credential-socket\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.190927 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.190877 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-socket\") pod \"7b584485-3ab6-4857-8701-7ec674cabe77\" (UID: \"7b584485-3ab6-4857-8701-7ec674cabe77\") " Oct 31 22:16:34.191223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.191195 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b584485-3ab6-4857-8701-7ec674cabe77-istiod-ca-cert" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:34.191433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.191411 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-certs" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.191578 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.191546 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.191578 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.191576 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b584485-3ab6-4857-8701-7ec674cabe77-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.191715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.191669 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-credential-socket" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.191816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.191794 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-socket" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.194018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.193996 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-istio-token" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:34.195358 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.195283 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-kube-api-access-5plww" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "kube-api-access-5plww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:34.195358 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.195345 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-envoy" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:34.198598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.198567 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b584485-3ab6-4857-8701-7ec674cabe77-istio-podinfo" pod "7b584485-3ab6-4857-8701-7ec674cabe77" (UID: "7b584485-3ab6-4857-8701-7ec674cabe77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:34.292794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.292766 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.293100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.292795 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.293100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.292805 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b584485-3ab6-4857-8701-7ec674cabe77-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.293100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.292813 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5plww\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-kube-api-access-5plww\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.293100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.292823 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b584485-3ab6-4857-8701-7ec674cabe77-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.293100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.292830 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.293100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.292838 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b584485-3ab6-4857-8701-7ec674cabe77-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:34.335330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.335285 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg"] Oct 31 22:16:34.335576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.335523 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" containerID="cri-o://8f9263da6efff226b32bee5b690331ef779113350f9f2fed0c1f4cec422c3aca" gracePeriod=30 Oct 31 22:16:34.552898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.552860 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" event={"ID":"71ce4284-2fc4-45e8-954e-596c3d17d73d","Type":"ContainerStarted","Data":"d52555c3ad3518dc56d5d74f9b651d147d7da79bcf4dd9b90a23e1c126fb28e3"} Oct 31 22:16:34.553059 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.552921 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" event={"ID":"71ce4284-2fc4-45e8-954e-596c3d17d73d","Type":"ContainerStarted","Data":"b23ed35c2da25c908871c2567f2e0c0dffa6143eeb8d34da6f88e304d2310d32"} Oct 31 22:16:34.554516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.554492 4294 generic.go:356] "Generic (PLEG): container finished" podID="7b584485-3ab6-4857-8701-7ec674cabe77" containerID="9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d" exitCode=0 Oct 31 22:16:34.554659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.554557 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" Oct 31 22:16:34.554659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.554568 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" event={"ID":"7b584485-3ab6-4857-8701-7ec674cabe77","Type":"ContainerDied","Data":"9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d"} Oct 31 22:16:34.554659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.554612 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf" event={"ID":"7b584485-3ab6-4857-8701-7ec674cabe77","Type":"ContainerDied","Data":"949a9012a3e5d5ba6a6511f86e1c853febd9bafa2e8fd73632574af9cdce56a8"} Oct 31 22:16:34.554659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.554634 4294 scope.go:122] "RemoveContainer" containerID="9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d" Oct 31 22:16:34.565515 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.565496 4294 scope.go:122] "RemoveContainer" containerID="9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d" Oct 31 22:16:34.565781 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:16:34.565763 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d\": container with ID starting with 9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d not found: ID does not exist" containerID="9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d" Oct 31 22:16:34.565831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.565791 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d"} err="failed to get container status \"9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d\": rpc error: code = NotFound desc = could not find container \"9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d\": container with ID starting with 9f3b4f2a3398e99b23f17d2c678a08f16b552e20c5c02f3448bdfb586b97400d not found: ID does not exist" Oct 31 22:16:34.596151 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.596115 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" podStartSLOduration=1.596103207 podStartE2EDuration="1.596103207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:16:34.593518536 +0000 UTC m=+3634.718625780" watchObservedRunningTime="2026-10-31 22:16:34.596103207 +0000 UTC m=+3634.721210482" Oct 31 22:16:34.631902 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.631873 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf"] Oct 31 22:16:34.638879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:34.638847 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-5drlf"] Oct 31 22:16:35.563905 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:35.563872 4294 generic.go:356] "Generic (PLEG): container finished" podID="71ce4284-2fc4-45e8-954e-596c3d17d73d" containerID="d52555c3ad3518dc56d5d74f9b651d147d7da79bcf4dd9b90a23e1c126fb28e3" exitCode=0 Oct 31 22:16:35.564295 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:35.563959 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" event={"ID":"71ce4284-2fc4-45e8-954e-596c3d17d73d","Type":"ContainerDied","Data":"d52555c3ad3518dc56d5d74f9b651d147d7da79bcf4dd9b90a23e1c126fb28e3"} Oct 31 22:16:35.985125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:35.985098 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" path="/var/lib/kubelet/pods/7b584485-3ab6-4857-8701-7ec674cabe77/volumes" Oct 31 22:16:36.613875 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.613394 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn"] Oct 31 22:16:36.614408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.614106 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" containerID="cri-o://3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214" gracePeriod=30 Oct 31 22:16:36.851633 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.851606 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:36.935036 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.935004 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-openshift-ca\") pod \"71ce4284-2fc4-45e8-954e-596c3d17d73d\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " Oct 31 22:16:36.935219 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.935201 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71ce4284-2fc4-45e8-954e-596c3d17d73d-kube-api-access-8kszr\" (UniqueName: \"kubernetes.io/projected/71ce4284-2fc4-45e8-954e-596c3d17d73d-kube-api-access-8kszr\") pod \"71ce4284-2fc4-45e8-954e-596c3d17d73d\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " Oct 31 22:16:36.935335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.935290 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-xks-ca\") pod \"71ce4284-2fc4-45e8-954e-596c3d17d73d\" (UID: \"71ce4284-2fc4-45e8-954e-596c3d17d73d\") " Oct 31 22:16:36.935460 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.935438 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-openshift-ca" pod "71ce4284-2fc4-45e8-954e-596c3d17d73d" (UID: "71ce4284-2fc4-45e8-954e-596c3d17d73d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:36.935756 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.935736 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:36.937377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.937282 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-xks-ca" pod "71ce4284-2fc4-45e8-954e-596c3d17d73d" (UID: "71ce4284-2fc4-45e8-954e-596c3d17d73d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:36.937377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:36.937313 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71ce4284-2fc4-45e8-954e-596c3d17d73d-kube-api-access-8kszr" pod "71ce4284-2fc4-45e8-954e-596c3d17d73d" (UID: "71ce4284-2fc4-45e8-954e-596c3d17d73d"). InnerVolumeSpecName "kube-api-access-8kszr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:37.036979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.036951 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kszr\" (UniqueName: \"kubernetes.io/projected/71ce4284-2fc4-45e8-954e-596c3d17d73d-kube-api-access-8kszr\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:37.036979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.036976 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71ce4284-2fc4-45e8-954e-596c3d17d73d-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:37.176395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.176327 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87"] Oct 31 22:16:37.176882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.176869 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" Oct 31 22:16:37.176922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.176883 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" Oct 31 22:16:37.176922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.176914 4294 cpu_manager.go:414] "containerMap: removing container" podUID="71ce4284-2fc4-45e8-954e-596c3d17d73d" containerName="revoke-keys" Oct 31 22:16:37.176922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.176920 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71ce4284-2fc4-45e8-954e-596c3d17d73d" containerName="revoke-keys" Oct 31 22:16:37.177021 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.177001 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="71ce4284-2fc4-45e8-954e-596c3d17d73d" containerName="revoke-keys" Oct 31 22:16:37.177021 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.177016 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b584485-3ab6-4857-8701-7ec674cabe77" containerName="istio-proxy" Oct 31 22:16:37.180425 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.180406 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.199190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.199141 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87"] Oct 31 22:16:37.238643 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.238611 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.238930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.238899 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.239050 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.239026 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjphf\" (UniqueName: \"kubernetes.io/projected/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-kube-api-access-vjphf\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.340436 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.340405 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjphf\" (UniqueName: \"kubernetes.io/projected/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-kube-api-access-vjphf\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.340563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.340506 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.340671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.340657 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.341247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.341223 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.342716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.342696 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.363466 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.363447 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjphf\" (UniqueName: \"kubernetes.io/projected/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-kube-api-access-vjphf\") pod \"maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.495686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.495660 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:37.580421 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.580400 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" Oct 31 22:16:37.580588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.580395 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v" event={"ID":"71ce4284-2fc4-45e8-954e-596c3d17d73d","Type":"ContainerDied","Data":"b23ed35c2da25c908871c2567f2e0c0dffa6143eeb8d34da6f88e304d2310d32"} Oct 31 22:16:37.580588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.580543 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b23ed35c2da25c908871c2567f2e0c0dffa6143eeb8d34da6f88e304d2310d32" Oct 31 22:16:37.795905 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:37.795877 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87"] Oct 31 22:16:37.796804 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:16:37.796781 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7f5db8_23de_4aa4_9c30_5f4cd1439471.slice/crio-019feea2a4e412f102bbdc0cf459bb7cc0ab0fa1efd57032ae0e1071e58a3017 WatchSource:0}: Error finding container 019feea2a4e412f102bbdc0cf459bb7cc0ab0fa1efd57032ae0e1071e58a3017: Status 404 returned error can't find the container with id 019feea2a4e412f102bbdc0cf459bb7cc0ab0fa1efd57032ae0e1071e58a3017 Oct 31 22:16:38.588864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:38.588824 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" event={"ID":"4f7f5db8-23de-4aa4-9c30-5f4cd1439471","Type":"ContainerStarted","Data":"1c2e47fc81f24fa0769b65f5bcdca9b7cf4b3a02102ab257980d0d3098b12831"} Oct 31 22:16:38.589041 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:38.588907 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" event={"ID":"4f7f5db8-23de-4aa4-9c30-5f4cd1439471","Type":"ContainerStarted","Data":"019feea2a4e412f102bbdc0cf459bb7cc0ab0fa1efd57032ae0e1071e58a3017"} Oct 31 22:16:38.631127 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:38.631073 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" podStartSLOduration=1.63105721 podStartE2EDuration="1.63105721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:16:38.629439508 +0000 UTC m=+3638.754546740" watchObservedRunningTime="2026-10-31 22:16:38.63105721 +0000 UTC m=+3638.756164440" Oct 31 22:16:39.599392 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.599324 4294 generic.go:356] "Generic (PLEG): container finished" podID="4f7f5db8-23de-4aa4-9c30-5f4cd1439471" containerID="1c2e47fc81f24fa0769b65f5bcdca9b7cf4b3a02102ab257980d0d3098b12831" exitCode=0 Oct 31 22:16:39.599392 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.599343 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" event={"ID":"4f7f5db8-23de-4aa4-9c30-5f4cd1439471","Type":"ContainerDied","Data":"1c2e47fc81f24fa0769b65f5bcdca9b7cf4b3a02102ab257980d0d3098b12831"} Oct 31 22:16:39.600895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.600873 4294 generic.go:356] "Generic (PLEG): container finished" podID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerID="8f9263da6efff226b32bee5b690331ef779113350f9f2fed0c1f4cec422c3aca" exitCode=0 Oct 31 22:16:39.600986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.600939 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" event={"ID":"bfd00683-6e8f-483e-9d30-93501b4786b8","Type":"ContainerDied","Data":"8f9263da6efff226b32bee5b690331ef779113350f9f2fed0c1f4cec422c3aca"} Oct 31 22:16:39.600986 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.600975 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" event={"ID":"bfd00683-6e8f-483e-9d30-93501b4786b8","Type":"ContainerDied","Data":"a7a501435cc9265719dd808a2935e05c3f70303a43e93b748587f7279be3c395"} Oct 31 22:16:39.601055 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.600992 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7a501435cc9265719dd808a2935e05c3f70303a43e93b748587f7279be3c395" Oct 31 22:16:39.608993 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.608974 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:16:39.668681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.668654 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd00683-6e8f-483e-9d30-93501b4786b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd00683-6e8f-483e-9d30-93501b4786b8-istiod-ca-cert\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.668816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.668706 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-envoy\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.668816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.668740 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-socket\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.668956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.668821 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-credential-socket\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.669060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669012 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-kube-api-access-qrpgk\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-kube-api-access-qrpgk\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.669128 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669056 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-socket" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:39.669189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669149 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-token\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-token\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.669189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669161 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-credential-socket" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:39.669295 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669199 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd00683-6e8f-483e-9d30-93501b4786b8-istiod-ca-cert" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:39.669295 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669206 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-certs\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.669295 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669269 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-podinfo\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.669490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669462 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-certs" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:39.669547 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669496 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-data\") pod \"bfd00683-6e8f-483e-9d30-93501b4786b8\" (UID: \"bfd00683-6e8f-483e-9d30-93501b4786b8\") " Oct 31 22:16:39.670225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.669770 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-data" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:39.670225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.670130 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.670225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.670158 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.670225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.670174 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.670225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.670190 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.670225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.670202 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfd00683-6e8f-483e-9d30-93501b4786b8-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.673479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.673441 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-envoy" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:39.673479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.673458 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-podinfo" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:39.673629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.673485 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-token" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:39.673629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.673517 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-kube-api-access-qrpgk" pod "bfd00683-6e8f-483e-9d30-93501b4786b8" (UID: "bfd00683-6e8f-483e-9d30-93501b4786b8"). InnerVolumeSpecName "kube-api-access-qrpgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:39.771188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.771158 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrpgk\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-kube-api-access-qrpgk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.771188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.771189 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.771414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.771203 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:39.771414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:39.771214 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfd00683-6e8f-483e-9d30-93501b4786b8-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:40.607642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.607609 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg" Oct 31 22:16:40.682498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.682462 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg"] Oct 31 22:16:40.693750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.693623 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-943963-openshift-default-68fddd7bb5-h94kg"] Oct 31 22:16:40.769410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.769386 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:40.884961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.884893 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-openshift-ca\") pod \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " Oct 31 22:16:40.884961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.884941 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-xks-ca\") pod \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " Oct 31 22:16:40.885147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.884992 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-kube-api-access-vjphf\" (UniqueName: \"kubernetes.io/projected/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-kube-api-access-vjphf\") pod \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\" (UID: \"4f7f5db8-23de-4aa4-9c30-5f4cd1439471\") " Oct 31 22:16:40.885349 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.885294 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-openshift-ca" pod "4f7f5db8-23de-4aa4-9c30-5f4cd1439471" (UID: "4f7f5db8-23de-4aa4-9c30-5f4cd1439471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:40.885582 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.885560 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:40.887188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.887164 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-xks-ca" pod "4f7f5db8-23de-4aa4-9c30-5f4cd1439471" (UID: "4f7f5db8-23de-4aa4-9c30-5f4cd1439471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:40.887335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.887213 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-kube-api-access-vjphf" pod "4f7f5db8-23de-4aa4-9c30-5f4cd1439471" (UID: "4f7f5db8-23de-4aa4-9c30-5f4cd1439471"). InnerVolumeSpecName "kube-api-access-vjphf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:40.986880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.986854 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:40.986880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:40.986879 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjphf\" (UniqueName: \"kubernetes.io/projected/4f7f5db8-23de-4aa4-9c30-5f4cd1439471-kube-api-access-vjphf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:41.593910 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:41.593873 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" podUID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:16:41.616018 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:41.615989 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" event={"ID":"4f7f5db8-23de-4aa4-9c30-5f4cd1439471","Type":"ContainerDied","Data":"019feea2a4e412f102bbdc0cf459bb7cc0ab0fa1efd57032ae0e1071e58a3017"} Oct 31 22:16:41.616413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:41.616018 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87" Oct 31 22:16:41.616413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:41.616034 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019feea2a4e412f102bbdc0cf459bb7cc0ab0fa1efd57032ae0e1071e58a3017" Oct 31 22:16:41.918378 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:41.908881 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:16:41.986156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:41.986119 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" path="/var/lib/kubelet/pods/bfd00683-6e8f-483e-9d30-93501b4786b8/volumes" Oct 31 22:16:42.028250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028222 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-istio-token\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-istio-token\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028265 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-kube-api-access-hp5b4\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-kube-api-access-hp5b4\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028439 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-envoy\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028477 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-data\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028563 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-socket\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028594 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beffe07e-cd89-436c-bad9-914094c2c732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beffe07e-cd89-436c-bad9-914094c2c732-istio-podinfo\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028639 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beffe07e-cd89-436c-bad9-914094c2c732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beffe07e-cd89-436c-bad9-914094c2c732-istiod-ca-cert\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028704 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-certs\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028749 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-credential-socket\") pod \"beffe07e-cd89-436c-bad9-914094c2c732\" (UID: \"beffe07e-cd89-436c-bad9-914094c2c732\") " Oct 31 22:16:42.028977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.028918 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-data" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:42.029324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.029137 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-credential-socket" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:42.029388 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.029351 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-certs" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:42.029455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.029383 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-socket" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:42.029455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.029424 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beffe07e-cd89-436c-bad9-914094c2c732-istiod-ca-cert" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:42.029455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.029450 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.029673 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.029469 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.031472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.031429 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-envoy" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:16:42.031583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.031560 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beffe07e-cd89-436c-bad9-914094c2c732-istio-podinfo" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:16:42.031855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.031830 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-istio-token" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:42.032786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.032753 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-kube-api-access-hp5b4" pod "beffe07e-cd89-436c-bad9-914094c2c732" (UID: "beffe07e-cd89-436c-bad9-914094c2c732"). InnerVolumeSpecName "kube-api-access-hp5b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:42.130709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.130635 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.130709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.130661 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.130709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.130669 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp5b4\" (UniqueName: \"kubernetes.io/projected/beffe07e-cd89-436c-bad9-914094c2c732-kube-api-access-hp5b4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.130709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.130679 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.130709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.130687 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beffe07e-cd89-436c-bad9-914094c2c732-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.130709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.130694 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beffe07e-cd89-436c-bad9-914094c2c732-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.130709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.130703 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beffe07e-cd89-436c-bad9-914094c2c732-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:42.623743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.623712 4294 generic.go:356] "Generic (PLEG): container finished" podID="beffe07e-cd89-436c-bad9-914094c2c732" containerID="3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214" exitCode=0 Oct 31 22:16:42.624190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.623777 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" Oct 31 22:16:42.624190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.623793 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" event={"ID":"beffe07e-cd89-436c-bad9-914094c2c732","Type":"ContainerDied","Data":"3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214"} Oct 31 22:16:42.624190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.623855 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" event={"ID":"beffe07e-cd89-436c-bad9-914094c2c732","Type":"ContainerDied","Data":"48bcedd9f71ddfdd0355d7d66c51435026fc3fb6c6032738a37371d8dd3b29c4"} Oct 31 22:16:42.624190 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.623885 4294 scope.go:122] "RemoveContainer" containerID="3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214" Oct 31 22:16:42.635006 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.634985 4294 scope.go:122] "RemoveContainer" containerID="3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214" Oct 31 22:16:42.635284 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:16:42.635263 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214\": container with ID starting with 3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214 not found: ID does not exist" containerID="3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214" Oct 31 22:16:42.635373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.635292 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214"} err="failed to get container status \"3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214\": rpc error: code = NotFound desc = could not find container \"3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214\": container with ID starting with 3d3afe1257702d53ffec3125dcf30dae6b198919e48b115e83140e80ace43214 not found: ID does not exist" Oct 31 22:16:42.678877 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.678822 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn"] Oct 31 22:16:42.690598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.690575 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn"] Oct 31 22:16:42.794020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.793987 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 22:16:42.794181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:42.794055 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c2d1b8-openshift-default-584f96b4df-vg5rn" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 22:16:43.985672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:43.985637 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beffe07e-cd89-436c-bad9-914094c2c732" path="/var/lib/kubelet/pods/beffe07e-cd89-436c-bad9-914094c2c732/volumes" Oct 31 22:16:44.716050 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716018 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb"] Oct 31 22:16:44.716577 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716563 4294 cpu_manager.go:414] "containerMap: removing container" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" Oct 31 22:16:44.716629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716578 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" Oct 31 22:16:44.716629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716596 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4f7f5db8-23de-4aa4-9c30-5f4cd1439471" containerName="revoke-keys" Oct 31 22:16:44.716629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716602 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7f5db8-23de-4aa4-9c30-5f4cd1439471" containerName="revoke-keys" Oct 31 22:16:44.716629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716627 4294 cpu_manager.go:414] "containerMap: removing container" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" Oct 31 22:16:44.716758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716632 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" Oct 31 22:16:44.716758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716704 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="beffe07e-cd89-436c-bad9-914094c2c732" containerName="istio-proxy" Oct 31 22:16:44.716758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716716 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7f5db8-23de-4aa4-9c30-5f4cd1439471" containerName="revoke-keys" Oct 31 22:16:44.716758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.716726 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd00683-6e8f-483e-9d30-93501b4786b8" containerName="istio-proxy" Oct 31 22:16:44.721718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.721699 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.739000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.738972 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb"] Oct 31 22:16:44.746391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.746365 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:16:44.746543 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.746500 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:16:44.867058 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.867019 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.867254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.867224 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.867330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.867292 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6fg\" (UniqueName: \"kubernetes.io/projected/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-kube-api-access-xl6fg\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.968038 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.967960 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.968038 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.968028 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6fg\" (UniqueName: \"kubernetes.io/projected/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-kube-api-access-xl6fg\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.968240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.968129 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.968626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.968601 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.970381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.970363 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:44.991412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:44.991385 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6fg\" (UniqueName: \"kubernetes.io/projected/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-kube-api-access-xl6fg\") pod \"maas-api-revoke-sub-05291d46-k4pdb\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:45.035758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:45.035737 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:45.347705 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:45.347682 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb"] Oct 31 22:16:45.349165 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:16:45.349137 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec6fd83_e9bd_47e7_8779_e41f41d736e8.slice/crio-82a4116cb4f2e31f3738195b8405a05086df520023f1dcdde3bc52a75bf2c07e WatchSource:0}: Error finding container 82a4116cb4f2e31f3738195b8405a05086df520023f1dcdde3bc52a75bf2c07e: Status 404 returned error can't find the container with id 82a4116cb4f2e31f3738195b8405a05086df520023f1dcdde3bc52a75bf2c07e Oct 31 22:16:45.649156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:45.649085 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" event={"ID":"7ec6fd83-e9bd-47e7-8779-e41f41d736e8","Type":"ContainerStarted","Data":"82a4116cb4f2e31f3738195b8405a05086df520023f1dcdde3bc52a75bf2c07e"} Oct 31 22:16:46.671843 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:46.671741 4294 generic.go:356] "Generic (PLEG): container finished" podID="7ec6fd83-e9bd-47e7-8779-e41f41d736e8" containerID="a6c0bc8d9a5d6a14c0fbdc75b7b0399d7cf8c4518c956fd6ae1d66058bc3c9e3" exitCode=0 Oct 31 22:16:46.672338 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:46.671962 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" event={"ID":"7ec6fd83-e9bd-47e7-8779-e41f41d736e8","Type":"ContainerDied","Data":"a6c0bc8d9a5d6a14c0fbdc75b7b0399d7cf8c4518c956fd6ae1d66058bc3c9e3"} Oct 31 22:16:47.846647 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:47.846623 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:48.012529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.012489 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-xks-ca\") pod \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " Oct 31 22:16:48.012681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.012608 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-kube-api-access-xl6fg\" (UniqueName: \"kubernetes.io/projected/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-kube-api-access-xl6fg\") pod \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " Oct 31 22:16:48.012755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.012737 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-openshift-ca\") pod \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\" (UID: \"7ec6fd83-e9bd-47e7-8779-e41f41d736e8\") " Oct 31 22:16:48.013208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.013177 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-openshift-ca" pod "7ec6fd83-e9bd-47e7-8779-e41f41d736e8" (UID: "7ec6fd83-e9bd-47e7-8779-e41f41d736e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:48.014501 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.014471 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-xks-ca" pod "7ec6fd83-e9bd-47e7-8779-e41f41d736e8" (UID: "7ec6fd83-e9bd-47e7-8779-e41f41d736e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:16:48.014651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.014633 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-kube-api-access-xl6fg" pod "7ec6fd83-e9bd-47e7-8779-e41f41d736e8" (UID: "7ec6fd83-e9bd-47e7-8779-e41f41d736e8"). InnerVolumeSpecName "kube-api-access-xl6fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:48.114879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.114787 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6fg\" (UniqueName: \"kubernetes.io/projected/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-kube-api-access-xl6fg\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:48.114879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.114820 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:48.114879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.114836 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ec6fd83-e9bd-47e7-8779-e41f41d736e8-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:48.688433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.688406 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" Oct 31 22:16:48.688605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.688434 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb" event={"ID":"7ec6fd83-e9bd-47e7-8779-e41f41d736e8","Type":"ContainerDied","Data":"82a4116cb4f2e31f3738195b8405a05086df520023f1dcdde3bc52a75bf2c07e"} Oct 31 22:16:48.688605 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:48.688485 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82a4116cb4f2e31f3738195b8405a05086df520023f1dcdde3bc52a75bf2c07e" Oct 31 22:16:49.190410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.190385 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 22:16:49.230034 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.229999 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/583e6add-c5b4-42b6-a9af-5de6e8563a8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e6add-c5b4-42b6-a9af-5de6e8563a8c-plugins-config-volume\") pod \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " Oct 31 22:16:49.230199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.230132 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/583e6add-c5b4-42b6-a9af-5de6e8563a8c-kube-api-access-mvqj5\" (UniqueName: \"kubernetes.io/projected/583e6add-c5b4-42b6-a9af-5de6e8563a8c-kube-api-access-mvqj5\") pod \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\" (UID: \"583e6add-c5b4-42b6-a9af-5de6e8563a8c\") " Oct 31 22:16:49.230498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.230464 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/583e6add-c5b4-42b6-a9af-5de6e8563a8c-plugins-config-volume" pod "583e6add-c5b4-42b6-a9af-5de6e8563a8c" (UID: "583e6add-c5b4-42b6-a9af-5de6e8563a8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:16:49.230923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.230897 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/583e6add-c5b4-42b6-a9af-5de6e8563a8c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.232606 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.232568 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583e6add-c5b4-42b6-a9af-5de6e8563a8c-kube-api-access-mvqj5" pod "583e6add-c5b4-42b6-a9af-5de6e8563a8c" (UID: "583e6add-c5b4-42b6-a9af-5de6e8563a8c"). InnerVolumeSpecName "kube-api-access-mvqj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:16:49.332395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.332320 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvqj5\" (UniqueName: \"kubernetes.io/projected/583e6add-c5b4-42b6-a9af-5de6e8563a8c-kube-api-access-mvqj5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:16:49.698320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.698225 4294 generic.go:356] "Generic (PLEG): container finished" podID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" containerID="3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764" exitCode=0 Oct 31 22:16:49.698320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.698274 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" Oct 31 22:16:49.698535 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.698334 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" event={"ID":"583e6add-c5b4-42b6-a9af-5de6e8563a8c","Type":"ContainerDied","Data":"3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764"} Oct 31 22:16:49.698535 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.698397 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b" event={"ID":"583e6add-c5b4-42b6-a9af-5de6e8563a8c","Type":"ContainerDied","Data":"6cf5688e833567f040d0c82cab249d7abc30be71c6dd637a57dc80479a4bedab"} Oct 31 22:16:49.698535 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.698422 4294 scope.go:122] "RemoveContainer" containerID="3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764" Oct 31 22:16:49.711992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.711971 4294 scope.go:122] "RemoveContainer" containerID="3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764" Oct 31 22:16:49.712257 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:16:49.712234 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764\": container with ID starting with 3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764 not found: ID does not exist" containerID="3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764" Oct 31 22:16:49.712356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.712263 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764"} err="failed to get container status \"3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764\": rpc error: code = NotFound desc = could not find container \"3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764\": container with ID starting with 3ae9c04c82a3ed6b7423c85c79a9578292adb0fb8f1bda8d5139120f8760d764 not found: ID does not exist" Oct 31 22:16:49.753062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.753028 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b"] Oct 31 22:16:49.766433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.766403 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-44405d-855f4cf98f-p874b"] Oct 31 22:16:49.985983 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:49.985953 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" path="/var/lib/kubelet/pods/583e6add-c5b4-42b6-a9af-5de6e8563a8c/volumes" Oct 31 22:16:59.956177 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:59.956145 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc"] Oct 31 22:16:59.975472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:59.975430 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-44405d-lb7tc"] Oct 31 22:16:59.995499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:16:59.995470 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e114a5c1-a188-4f54-b365-9c5e4828431f" path="/var/lib/kubelet/pods/e114a5c1-a188-4f54-b365-9c5e4828431f/volumes" Oct 31 22:17:01.310735 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:17:01.310693 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:17:02.135854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:02.135823 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87"] Oct 31 22:17:02.147819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:02.147791 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-4bd51b-fcd87"] Oct 31 22:17:03.267909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.267875 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2"] Oct 31 22:17:03.534963 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.534874 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s"] Oct 31 22:17:03.535503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.535483 4294 cpu_manager.go:414] "containerMap: removing container" podUID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" containerName="payload-processing" Oct 31 22:17:03.535614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.535504 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" containerName="payload-processing" Oct 31 22:17:03.535614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.535595 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7ec6fd83-e9bd-47e7-8779-e41f41d736e8" containerName="revoke-keys" Oct 31 22:17:03.535614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.535604 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec6fd83-e9bd-47e7-8779-e41f41d736e8" containerName="revoke-keys" Oct 31 22:17:03.535809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.535759 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec6fd83-e9bd-47e7-8779-e41f41d736e8" containerName="revoke-keys" Oct 31 22:17:03.535809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.535779 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="583e6add-c5b4-42b6-a9af-5de6e8563a8c" containerName="payload-processing" Oct 31 22:17:03.540828 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.540805 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.557142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.557014 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:17:03.557142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.557114 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:17:03.557914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.557892 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s"] Oct 31 22:17:03.625142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.625108 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vskkt\" (UniqueName: \"kubernetes.io/projected/65186725-eaa4-4bf7-a632-b95820c14768-kube-api-access-vskkt\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.625351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.625178 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.625351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.625240 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.727017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.726983 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vskkt\" (UniqueName: \"kubernetes.io/projected/65186725-eaa4-4bf7-a632-b95820c14768-kube-api-access-vskkt\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.727183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.727086 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.727183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.727127 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.727783 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.727759 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.729339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.729317 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.750292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.750265 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vskkt\" (UniqueName: \"kubernetes.io/projected/65186725-eaa4-4bf7-a632-b95820c14768-kube-api-access-vskkt\") pod \"maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.854492 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.854423 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:03.913871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.913725 4294 scope.go:122] "RemoveContainer" containerID="163f5f9f21c25a219e5341c942bbf4b6664c9fd6982117918d18f2d8adc26a37" Oct 31 22:17:03.930517 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.929886 4294 scope.go:122] "RemoveContainer" containerID="bb1f4209bd5fd847092a80e75d9db7e826272545ee8510aa7c4099dd8e09a4dd" Oct 31 22:17:03.942374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.942255 4294 scope.go:122] "RemoveContainer" containerID="8f9263da6efff226b32bee5b690331ef779113350f9f2fed0c1f4cec422c3aca" Oct 31 22:17:03.952705 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.952687 4294 scope.go:122] "RemoveContainer" containerID="882e643115ddf8b7aac5e4ea4b8dc50b9d7ae25aeb180a1f8cadb7fd7df80c5d" Oct 31 22:17:03.986365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:03.986341 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7f5db8-23de-4aa4-9c30-5f4cd1439471" path="/var/lib/kubelet/pods/4f7f5db8-23de-4aa4-9c30-5f4cd1439471/volumes" Oct 31 22:17:04.159846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:04.159813 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s"] Oct 31 22:17:04.160030 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:17:04.160008 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65186725_eaa4_4bf7_a632_b95820c14768.slice/crio-15b87748c04b3af1e7d06f07229e3b18e864cae67fb66415f1bfa43bf9fda287 WatchSource:0}: Error finding container 15b87748c04b3af1e7d06f07229e3b18e864cae67fb66415f1bfa43bf9fda287: Status 404 returned error can't find the container with id 15b87748c04b3af1e7d06f07229e3b18e864cae67fb66415f1bfa43bf9fda287 Oct 31 22:17:04.813239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:04.813196 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" event={"ID":"65186725-eaa4-4bf7-a632-b95820c14768","Type":"ContainerStarted","Data":"6288db7e2d8d085cba46055ea869483fcdd0e21838183b0f15df899bd3793f56"} Oct 31 22:17:04.813835 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:04.813257 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" event={"ID":"65186725-eaa4-4bf7-a632-b95820c14768","Type":"ContainerStarted","Data":"15b87748c04b3af1e7d06f07229e3b18e864cae67fb66415f1bfa43bf9fda287"} Oct 31 22:17:04.857426 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:04.857377 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" podStartSLOduration=1.85736499 podStartE2EDuration="1.85736499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:17:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:17:04.853841225 +0000 UTC m=+3664.978948455" watchObservedRunningTime="2026-10-31 22:17:04.85736499 +0000 UTC m=+3664.982472221" Oct 31 22:17:05.821895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:05.821861 4294 generic.go:356] "Generic (PLEG): container finished" podID="65186725-eaa4-4bf7-a632-b95820c14768" containerID="6288db7e2d8d085cba46055ea869483fcdd0e21838183b0f15df899bd3793f56" exitCode=0 Oct 31 22:17:05.822357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:05.821949 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" event={"ID":"65186725-eaa4-4bf7-a632-b95820c14768","Type":"ContainerDied","Data":"6288db7e2d8d085cba46055ea869483fcdd0e21838183b0f15df899bd3793f56"} Oct 31 22:17:06.980223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:06.980196 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:07.075726 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.075690 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65186725-eaa4-4bf7-a632-b95820c14768-kube-api-access-vskkt\" (UniqueName: \"kubernetes.io/projected/65186725-eaa4-4bf7-a632-b95820c14768-kube-api-access-vskkt\") pod \"65186725-eaa4-4bf7-a632-b95820c14768\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " Oct 31 22:17:07.075915 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.075737 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-openshift-ca\") pod \"65186725-eaa4-4bf7-a632-b95820c14768\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " Oct 31 22:17:07.075991 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.075930 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-xks-ca\") pod \"65186725-eaa4-4bf7-a632-b95820c14768\" (UID: \"65186725-eaa4-4bf7-a632-b95820c14768\") " Oct 31 22:17:07.076067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.076050 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-openshift-ca" pod "65186725-eaa4-4bf7-a632-b95820c14768" (UID: "65186725-eaa4-4bf7-a632-b95820c14768"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:07.076607 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.076574 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:07.077982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.077956 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-xks-ca" pod "65186725-eaa4-4bf7-a632-b95820c14768" (UID: "65186725-eaa4-4bf7-a632-b95820c14768"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:07.077982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.077974 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65186725-eaa4-4bf7-a632-b95820c14768-kube-api-access-vskkt" pod "65186725-eaa4-4bf7-a632-b95820c14768" (UID: "65186725-eaa4-4bf7-a632-b95820c14768"). InnerVolumeSpecName "kube-api-access-vskkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:07.178405 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.178369 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vskkt\" (UniqueName: \"kubernetes.io/projected/65186725-eaa4-4bf7-a632-b95820c14768-kube-api-access-vskkt\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:07.178405 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.178404 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65186725-eaa4-4bf7-a632-b95820c14768-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:07.837229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.837189 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" event={"ID":"65186725-eaa4-4bf7-a632-b95820c14768","Type":"ContainerDied","Data":"15b87748c04b3af1e7d06f07229e3b18e864cae67fb66415f1bfa43bf9fda287"} Oct 31 22:17:07.837229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.837219 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s" Oct 31 22:17:07.837523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.837246 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15b87748c04b3af1e7d06f07229e3b18e864cae67fb66415f1bfa43bf9fda287" Oct 31 22:17:07.912439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.912410 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45"] Oct 31 22:17:07.912981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.912966 4294 cpu_manager.go:414] "containerMap: removing container" podUID="65186725-eaa4-4bf7-a632-b95820c14768" containerName="revoke-keys" Oct 31 22:17:07.913025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.912983 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65186725-eaa4-4bf7-a632-b95820c14768" containerName="revoke-keys" Oct 31 22:17:07.913093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.913084 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="65186725-eaa4-4bf7-a632-b95820c14768" containerName="revoke-keys" Oct 31 22:17:07.916233 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.916216 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:07.931209 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.931169 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:17:07.931372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.931272 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:17:07.934868 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:07.934845 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45"] Oct 31 22:17:08.016355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.016289 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.016815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.016551 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.016815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.016674 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqbb\" (UniqueName: \"kubernetes.io/projected/406c2372-7b78-422b-80bb-52622c8c23cf-kube-api-access-nrqbb\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.117939 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.117860 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.118084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.117936 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqbb\" (UniqueName: \"kubernetes.io/projected/406c2372-7b78-422b-80bb-52622c8c23cf-kube-api-access-nrqbb\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.118084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.117991 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.118664 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.118642 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.120294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.120271 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.141402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.141382 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqbb\" (UniqueName: \"kubernetes.io/projected/406c2372-7b78-422b-80bb-52622c8c23cf-kube-api-access-nrqbb\") pod \"maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.229747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.229711 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:08.541714 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.541622 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45"] Oct 31 22:17:08.544623 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:17:08.544575 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod406c2372_7b78_422b_80bb_52622c8c23cf.slice/crio-0ad63ce29142d59263cf700e473a3cab0ac4f54401be228b251013b81cfc0fbc WatchSource:0}: Error finding container 0ad63ce29142d59263cf700e473a3cab0ac4f54401be228b251013b81cfc0fbc: Status 404 returned error can't find the container with id 0ad63ce29142d59263cf700e473a3cab0ac4f54401be228b251013b81cfc0fbc Oct 31 22:17:08.848132 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:08.846867 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" event={"ID":"406c2372-7b78-422b-80bb-52622c8c23cf","Type":"ContainerStarted","Data":"0ad63ce29142d59263cf700e473a3cab0ac4f54401be228b251013b81cfc0fbc"} Oct 31 22:17:09.857815 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:09.857779 4294 generic.go:356] "Generic (PLEG): container finished" podID="406c2372-7b78-422b-80bb-52622c8c23cf" containerID="53bf670270def8bf906ab1703ab4c59378c5bb87a2ac9d6db98e2047c364a555" exitCode=0 Oct 31 22:17:09.858236 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:09.857870 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" event={"ID":"406c2372-7b78-422b-80bb-52622c8c23cf","Type":"ContainerDied","Data":"53bf670270def8bf906ab1703ab4c59378c5bb87a2ac9d6db98e2047c364a555"} Oct 31 22:17:10.396572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.396539 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w"] Oct 31 22:17:10.400417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.400394 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.419776 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.419748 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w"] Oct 31 22:17:10.551411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.551376 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.551602 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.551586 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.551659 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.551628 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzc7\" (UniqueName: \"kubernetes.io/projected/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-kube-api-access-wxzc7\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.663734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.653753 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.663734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.653806 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzc7\" (UniqueName: \"kubernetes.io/projected/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-kube-api-access-wxzc7\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.663734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.653985 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.663734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.655114 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.663734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.657565 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.677799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.677764 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzc7\" (UniqueName: \"kubernetes.io/projected/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-kube-api-access-wxzc7\") pod \"maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:10.714001 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:10.713971 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:11.001224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.001196 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:11.023187 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:17:11.023161 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e89236_e66e_4aa6_81fa_ce6c4a920ff3.slice/crio-c338cce85fb5cf43722ee0c514c09eee8299e90072028e6a9efb52fc9f3536e0 WatchSource:0}: Error finding container c338cce85fb5cf43722ee0c514c09eee8299e90072028e6a9efb52fc9f3536e0: Status 404 returned error can't find the container with id c338cce85fb5cf43722ee0c514c09eee8299e90072028e6a9efb52fc9f3536e0 Oct 31 22:17:11.032727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.032702 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w"] Oct 31 22:17:11.161246 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.161164 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-xks-ca\") pod \"406c2372-7b78-422b-80bb-52622c8c23cf\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " Oct 31 22:17:11.161417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.161277 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-openshift-ca\") pod \"406c2372-7b78-422b-80bb-52622c8c23cf\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " Oct 31 22:17:11.161417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.161388 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/406c2372-7b78-422b-80bb-52622c8c23cf-kube-api-access-nrqbb\" (UniqueName: \"kubernetes.io/projected/406c2372-7b78-422b-80bb-52622c8c23cf-kube-api-access-nrqbb\") pod \"406c2372-7b78-422b-80bb-52622c8c23cf\" (UID: \"406c2372-7b78-422b-80bb-52622c8c23cf\") " Oct 31 22:17:11.161656 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.161620 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-openshift-ca" pod "406c2372-7b78-422b-80bb-52622c8c23cf" (UID: "406c2372-7b78-422b-80bb-52622c8c23cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:11.161855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.161834 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:11.163364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.163341 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-xks-ca" pod "406c2372-7b78-422b-80bb-52622c8c23cf" (UID: "406c2372-7b78-422b-80bb-52622c8c23cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:11.163441 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.163420 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/406c2372-7b78-422b-80bb-52622c8c23cf-kube-api-access-nrqbb" pod "406c2372-7b78-422b-80bb-52622c8c23cf" (UID: "406c2372-7b78-422b-80bb-52622c8c23cf"). InnerVolumeSpecName "kube-api-access-nrqbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:11.262910 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.262884 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/406c2372-7b78-422b-80bb-52622c8c23cf-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:11.262910 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.262908 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqbb\" (UniqueName: \"kubernetes.io/projected/406c2372-7b78-422b-80bb-52622c8c23cf-kube-api-access-nrqbb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:11.884258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.884228 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" Oct 31 22:17:11.884469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.884257 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45" event={"ID":"406c2372-7b78-422b-80bb-52622c8c23cf","Type":"ContainerDied","Data":"0ad63ce29142d59263cf700e473a3cab0ac4f54401be228b251013b81cfc0fbc"} Oct 31 22:17:11.885106 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.885044 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ad63ce29142d59263cf700e473a3cab0ac4f54401be228b251013b81cfc0fbc" Oct 31 22:17:11.886503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.886347 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" event={"ID":"54e89236-e66e-4aa6-81fa-ce6c4a920ff3","Type":"ContainerStarted","Data":"6fe0ce58fc5b02e0a4d4291aeb4879b57471cdc95523b23d6d8795f29faed675"} Oct 31 22:17:11.886503 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:11.886390 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" event={"ID":"54e89236-e66e-4aa6-81fa-ce6c4a920ff3","Type":"ContainerStarted","Data":"c338cce85fb5cf43722ee0c514c09eee8299e90072028e6a9efb52fc9f3536e0"} Oct 31 22:17:12.654894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:12.654865 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6"] Oct 31 22:17:12.663508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:12.663479 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls"] Oct 31 22:17:12.674933 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:12.674903 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s"] Oct 31 22:17:12.894821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:12.894785 4294 generic.go:356] "Generic (PLEG): container finished" podID="54e89236-e66e-4aa6-81fa-ce6c4a920ff3" containerID="6fe0ce58fc5b02e0a4d4291aeb4879b57471cdc95523b23d6d8795f29faed675" exitCode=0 Oct 31 22:17:12.894978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:12.894872 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" event={"ID":"54e89236-e66e-4aa6-81fa-ce6c4a920ff3","Type":"ContainerDied","Data":"6fe0ce58fc5b02e0a4d4291aeb4879b57471cdc95523b23d6d8795f29faed675"} Oct 31 22:17:13.047721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.047688 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:13.193854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.193819 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-kube-api-access-wxzc7\" (UniqueName: \"kubernetes.io/projected/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-kube-api-access-wxzc7\") pod \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " Oct 31 22:17:13.194070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.193962 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-openshift-ca\") pod \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " Oct 31 22:17:13.194070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.194058 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-xks-ca\") pod \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\" (UID: \"54e89236-e66e-4aa6-81fa-ce6c4a920ff3\") " Oct 31 22:17:13.194389 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.194363 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-openshift-ca" pod "54e89236-e66e-4aa6-81fa-ce6c4a920ff3" (UID: "54e89236-e66e-4aa6-81fa-ce6c4a920ff3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:13.194739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.194702 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:13.196538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.196508 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-xks-ca" pod "54e89236-e66e-4aa6-81fa-ce6c4a920ff3" (UID: "54e89236-e66e-4aa6-81fa-ce6c4a920ff3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:13.196671 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.196569 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-kube-api-access-wxzc7" pod "54e89236-e66e-4aa6-81fa-ce6c4a920ff3" (UID: "54e89236-e66e-4aa6-81fa-ce6c4a920ff3"). InnerVolumeSpecName "kube-api-access-wxzc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:13.295990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.295950 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:13.295990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.295990 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxzc7\" (UniqueName: \"kubernetes.io/projected/54e89236-e66e-4aa6-81fa-ce6c4a920ff3-kube-api-access-wxzc7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:13.902961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.902929 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" Oct 31 22:17:13.902961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.902938 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w" event={"ID":"54e89236-e66e-4aa6-81fa-ce6c4a920ff3","Type":"ContainerDied","Data":"c338cce85fb5cf43722ee0c514c09eee8299e90072028e6a9efb52fc9f3536e0"} Oct 31 22:17:13.903500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:13.902992 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c338cce85fb5cf43722ee0c514c09eee8299e90072028e6a9efb52fc9f3536e0" Oct 31 22:17:14.006386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:14.006350 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6"] Oct 31 22:17:14.039900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:14.039869 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s"] Oct 31 22:17:18.291208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.291143 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" podUID="a77ca828-2884-4809-b2a7-9ead63055834" containerName="main" containerID="cri-o://ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b" gracePeriod=45 Oct 31 22:17:18.600073 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.600039 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:17:18.769700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.769662 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-tmp-dir\") pod \"a77ca828-2884-4809-b2a7-9ead63055834\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " Oct 31 22:17:18.769903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.769716 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77ca828-2884-4809-b2a7-9ead63055834-kube-api-access-ppdf7\" (UniqueName: \"kubernetes.io/projected/a77ca828-2884-4809-b2a7-9ead63055834-kube-api-access-ppdf7\") pod \"a77ca828-2884-4809-b2a7-9ead63055834\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " Oct 31 22:17:18.769903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.769794 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a77ca828-2884-4809-b2a7-9ead63055834-tls-certs\" (UniqueName: \"kubernetes.io/secret/a77ca828-2884-4809-b2a7-9ead63055834-tls-certs\") pod \"a77ca828-2884-4809-b2a7-9ead63055834\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " Oct 31 22:17:18.769903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.769865 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-dshm\") pod \"a77ca828-2884-4809-b2a7-9ead63055834\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " Oct 31 22:17:18.770089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.769910 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-model-cache\") pod \"a77ca828-2884-4809-b2a7-9ead63055834\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " Oct 31 22:17:18.770089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.769994 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-home\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-home\") pod \"a77ca828-2884-4809-b2a7-9ead63055834\" (UID: \"a77ca828-2884-4809-b2a7-9ead63055834\") " Oct 31 22:17:18.770089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.770039 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-tmp-dir" pod "a77ca828-2884-4809-b2a7-9ead63055834" (UID: "a77ca828-2884-4809-b2a7-9ead63055834"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:18.770254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.770229 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-model-cache" pod "a77ca828-2884-4809-b2a7-9ead63055834" (UID: "a77ca828-2884-4809-b2a7-9ead63055834"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:18.770374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.770339 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-home" pod "a77ca828-2884-4809-b2a7-9ead63055834" (UID: "a77ca828-2884-4809-b2a7-9ead63055834"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:18.770552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.770535 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:18.770624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.770557 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:18.770624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.770571 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:18.772254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.772228 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-dshm" pod "a77ca828-2884-4809-b2a7-9ead63055834" (UID: "a77ca828-2884-4809-b2a7-9ead63055834"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:18.772393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.772246 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77ca828-2884-4809-b2a7-9ead63055834-kube-api-access-ppdf7" pod "a77ca828-2884-4809-b2a7-9ead63055834" (UID: "a77ca828-2884-4809-b2a7-9ead63055834"). InnerVolumeSpecName "kube-api-access-ppdf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:18.772393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.772313 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77ca828-2884-4809-b2a7-9ead63055834-tls-certs" pod "a77ca828-2884-4809-b2a7-9ead63055834" (UID: "a77ca828-2884-4809-b2a7-9ead63055834"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:18.871678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.871605 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppdf7\" (UniqueName: \"kubernetes.io/projected/a77ca828-2884-4809-b2a7-9ead63055834-kube-api-access-ppdf7\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:18.871678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.871633 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a77ca828-2884-4809-b2a7-9ead63055834-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:18.871678 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.871642 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a77ca828-2884-4809-b2a7-9ead63055834-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:18.934485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.934452 4294 generic.go:356] "Generic (PLEG): container finished" podID="a77ca828-2884-4809-b2a7-9ead63055834" containerID="ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b" exitCode=0 Oct 31 22:17:18.934662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.934501 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" event={"ID":"a77ca828-2884-4809-b2a7-9ead63055834","Type":"ContainerDied","Data":"ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b"} Oct 31 22:17:18.934662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.934532 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" event={"ID":"a77ca828-2884-4809-b2a7-9ead63055834","Type":"ContainerDied","Data":"5fb7ae5fc6772e3f15cbde022c15324e5ec62f62941f9af5fe37e655da0d06ec"} Oct 31 22:17:18.934662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.934547 4294 scope.go:122] "RemoveContainer" containerID="ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b" Oct 31 22:17:18.934662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.934505 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2" Oct 31 22:17:18.945475 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.945457 4294 scope.go:122] "RemoveContainer" containerID="ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b" Oct 31 22:17:18.945772 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:17:18.945753 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b\": container with ID starting with ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b not found: ID does not exist" containerID="ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b" Oct 31 22:17:18.945833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.945780 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b"} err="failed to get container status \"ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b\": rpc error: code = NotFound desc = could not find container \"ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b\": container with ID starting with ea1434ad4e64c9c6a6733ba0a6d158d554a2390368d7c85253aac72fa986ff4b not found: ID does not exist" Oct 31 22:17:18.987625 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.987430 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2"] Oct 31 22:17:18.991550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:18.991522 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c08360/test-model-c08360-kserve-7d8985c679-lnqz2"] Oct 31 22:17:19.006600 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:19.006567 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" podUID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" containerName="payload-processing" containerID="cri-o://86151cacf887b641f5139e4eaaa8690e0a6fc291b1959205f267e212417220d5" gracePeriod=25 Oct 31 22:17:19.051386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:19.040402 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" podUID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" containerName="payload-pre-processing" containerID="cri-o://6ebcbf6ce44f0ba414255b80a4765ba0fe169ced5318e75ecec533d23dbd700f" gracePeriod=25 Oct 31 22:17:19.984727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:19.984693 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77ca828-2884-4809-b2a7-9ead63055834" path="/var/lib/kubelet/pods/a77ca828-2884-4809-b2a7-9ead63055834/volumes" Oct 31 22:17:22.055168 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.055127 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk"] Oct 31 22:17:22.055880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.055857 4294 cpu_manager.go:414] "containerMap: removing container" podUID="54e89236-e66e-4aa6-81fa-ce6c4a920ff3" containerName="revoke-keys" Oct 31 22:17:22.055980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.055881 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e89236-e66e-4aa6-81fa-ce6c4a920ff3" containerName="revoke-keys" Oct 31 22:17:22.055980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.055918 4294 cpu_manager.go:414] "containerMap: removing container" podUID="406c2372-7b78-422b-80bb-52622c8c23cf" containerName="revoke-keys" Oct 31 22:17:22.055980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.055926 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="406c2372-7b78-422b-80bb-52622c8c23cf" containerName="revoke-keys" Oct 31 22:17:22.055980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.055945 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a77ca828-2884-4809-b2a7-9ead63055834" containerName="main" Oct 31 22:17:22.055980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.055952 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77ca828-2884-4809-b2a7-9ead63055834" containerName="main" Oct 31 22:17:22.056241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.056105 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e89236-e66e-4aa6-81fa-ce6c4a920ff3" containerName="revoke-keys" Oct 31 22:17:22.056241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.056123 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="406c2372-7b78-422b-80bb-52622c8c23cf" containerName="revoke-keys" Oct 31 22:17:22.056241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.056135 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77ca828-2884-4809-b2a7-9ead63055834" containerName="main" Oct 31 22:17:22.061057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.061036 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.078476 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.078448 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv"] Oct 31 22:17:22.083002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.082976 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk"] Oct 31 22:17:22.083150 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.083122 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.085254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.085229 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:17:22.086063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.085688 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:17:22.101240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.101206 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv"] Oct 31 22:17:22.120539 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.120496 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.120725 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.120639 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wljgn\" (UniqueName: \"kubernetes.io/projected/ce717cad-91fe-441e-b3ad-f6fff5d888fa-kube-api-access-wljgn\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.120725 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.120685 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.120725 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.120705 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.120942 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.120911 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrr5z\" (UniqueName: \"kubernetes.io/projected/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-kube-api-access-xrr5z\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.121030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.121012 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.222629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.222595 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.222823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.222652 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.222823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.222724 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wljgn\" (UniqueName: \"kubernetes.io/projected/ce717cad-91fe-441e-b3ad-f6fff5d888fa-kube-api-access-wljgn\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.222823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.222756 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.222823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.222776 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.223043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.222838 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrr5z\" (UniqueName: \"kubernetes.io/projected/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-kube-api-access-xrr5z\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.223433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.223407 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.223552 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.223446 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.225134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.225111 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.225247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.225224 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.248465 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.248442 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wljgn\" (UniqueName: \"kubernetes.io/projected/ce717cad-91fe-441e-b3ad-f6fff5d888fa-kube-api-access-wljgn\") pod \"maas-api-revoke-sub-671d8dec-hcqtk\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.248589 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.248465 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrr5z\" (UniqueName: \"kubernetes.io/projected/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-kube-api-access-xrr5z\") pod \"maas-api-revoke-sub-a8e1165d-wtbhv\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.376196 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.376116 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:22.400477 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.400448 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:22.682719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.682691 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk"] Oct 31 22:17:22.683928 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:17:22.683904 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce717cad_91fe_441e_b3ad_f6fff5d888fa.slice/crio-b32a6b1eb6a2ee449cd631468653f34af440bff66ec1474fb27873c66d421bda WatchSource:0}: Error finding container b32a6b1eb6a2ee449cd631468653f34af440bff66ec1474fb27873c66d421bda: Status 404 returned error can't find the container with id b32a6b1eb6a2ee449cd631468653f34af440bff66ec1474fb27873c66d421bda Oct 31 22:17:22.737663 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.737635 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv"] Oct 31 22:17:22.738510 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:17:22.738483 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fcd8c4f_3298_4c2d_82a7_325ca715c9ac.slice/crio-85e10fade7d57176306250c887fde8064c53c53987cdc82258744ec1d9dba575 WatchSource:0}: Error finding container 85e10fade7d57176306250c887fde8064c53c53987cdc82258744ec1d9dba575: Status 404 returned error can't find the container with id 85e10fade7d57176306250c887fde8064c53c53987cdc82258744ec1d9dba575 Oct 31 22:17:22.965315 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.965216 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" event={"ID":"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac","Type":"ContainerStarted","Data":"85e10fade7d57176306250c887fde8064c53c53987cdc82258744ec1d9dba575"} Oct 31 22:17:22.966798 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:22.966769 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" event={"ID":"ce717cad-91fe-441e-b3ad-f6fff5d888fa","Type":"ContainerStarted","Data":"b32a6b1eb6a2ee449cd631468653f34af440bff66ec1474fb27873c66d421bda"} Oct 31 22:17:23.977092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:23.977000 4294 generic.go:356] "Generic (PLEG): container finished" podID="0fcd8c4f-3298-4c2d-82a7-325ca715c9ac" containerID="16f545efab3abba263f3ce9db47dec567a3f2d373e3c0a585f32887ab01f8700" exitCode=0 Oct 31 22:17:23.979016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:23.978983 4294 generic.go:356] "Generic (PLEG): container finished" podID="ce717cad-91fe-441e-b3ad-f6fff5d888fa" containerID="75d912ffc1274546627df32c2388fae68a5680a99c25630b780c0b1c42bc02b2" exitCode=0 Oct 31 22:17:23.985623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:23.985584 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" event={"ID":"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac","Type":"ContainerDied","Data":"16f545efab3abba263f3ce9db47dec567a3f2d373e3c0a585f32887ab01f8700"} Oct 31 22:17:23.985777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:23.985647 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" event={"ID":"ce717cad-91fe-441e-b3ad-f6fff5d888fa","Type":"ContainerDied","Data":"75d912ffc1274546627df32c2388fae68a5680a99c25630b780c0b1c42bc02b2"} Oct 31 22:17:24.649817 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:24.649779 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s"] Oct 31 22:17:24.653139 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:24.653109 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-cd421c-f9h7s"] Oct 31 22:17:25.140484 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.140458 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:25.194497 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.194469 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:25.233039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.233008 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-openshift-ca\") pod \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " Oct 31 22:17:25.233220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.233141 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce717cad-91fe-441e-b3ad-f6fff5d888fa-kube-api-access-wljgn\" (UniqueName: \"kubernetes.io/projected/ce717cad-91fe-441e-b3ad-f6fff5d888fa-kube-api-access-wljgn\") pod \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " Oct 31 22:17:25.233353 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.233208 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-xks-ca\") pod \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\" (UID: \"ce717cad-91fe-441e-b3ad-f6fff5d888fa\") " Oct 31 22:17:25.233990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.233499 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-openshift-ca" pod "ce717cad-91fe-441e-b3ad-f6fff5d888fa" (UID: "ce717cad-91fe-441e-b3ad-f6fff5d888fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:25.233990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.233910 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:25.235856 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.235781 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce717cad-91fe-441e-b3ad-f6fff5d888fa-kube-api-access-wljgn" pod "ce717cad-91fe-441e-b3ad-f6fff5d888fa" (UID: "ce717cad-91fe-441e-b3ad-f6fff5d888fa"). InnerVolumeSpecName "kube-api-access-wljgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:25.239393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.239363 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-xks-ca" pod "ce717cad-91fe-441e-b3ad-f6fff5d888fa" (UID: "ce717cad-91fe-441e-b3ad-f6fff5d888fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:25.334709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.334679 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-openshift-ca\") pod \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " Oct 31 22:17:25.334847 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.334831 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-kube-api-access-xrr5z\" (UniqueName: \"kubernetes.io/projected/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-kube-api-access-xrr5z\") pod \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " Oct 31 22:17:25.334962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.334943 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-xks-ca\") pod \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\" (UID: \"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac\") " Oct 31 22:17:25.335067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.335049 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-openshift-ca" pod "0fcd8c4f-3298-4c2d-82a7-325ca715c9ac" (UID: "0fcd8c4f-3298-4c2d-82a7-325ca715c9ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:25.335420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.335395 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wljgn\" (UniqueName: \"kubernetes.io/projected/ce717cad-91fe-441e-b3ad-f6fff5d888fa-kube-api-access-wljgn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:25.335420 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.335421 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce717cad-91fe-441e-b3ad-f6fff5d888fa-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:25.335575 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.335430 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:25.336820 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.336796 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-kube-api-access-xrr5z" pod "0fcd8c4f-3298-4c2d-82a7-325ca715c9ac" (UID: "0fcd8c4f-3298-4c2d-82a7-325ca715c9ac"). InnerVolumeSpecName "kube-api-access-xrr5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:25.336889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.336871 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-xks-ca" pod "0fcd8c4f-3298-4c2d-82a7-325ca715c9ac" (UID: "0fcd8c4f-3298-4c2d-82a7-325ca715c9ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:25.436418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.436386 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrr5z\" (UniqueName: \"kubernetes.io/projected/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-kube-api-access-xrr5z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:25.436418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.436414 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fcd8c4f-3298-4c2d-82a7-325ca715c9ac-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:25.985504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.985468 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65186725-eaa4-4bf7-a632-b95820c14768" path="/var/lib/kubelet/pods/65186725-eaa4-4bf7-a632-b95820c14768/volumes" Oct 31 22:17:25.994317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.994284 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" Oct 31 22:17:25.994461 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.994330 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8e1165d-wtbhv" event={"ID":"0fcd8c4f-3298-4c2d-82a7-325ca715c9ac","Type":"ContainerDied","Data":"85e10fade7d57176306250c887fde8064c53c53987cdc82258744ec1d9dba575"} Oct 31 22:17:25.994461 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.994375 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e10fade7d57176306250c887fde8064c53c53987cdc82258744ec1d9dba575" Oct 31 22:17:25.995891 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.995871 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" Oct 31 22:17:25.996016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.995874 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671d8dec-hcqtk" event={"ID":"ce717cad-91fe-441e-b3ad-f6fff5d888fa","Type":"ContainerDied","Data":"b32a6b1eb6a2ee449cd631468653f34af440bff66ec1474fb27873c66d421bda"} Oct 31 22:17:25.996016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:25.995989 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b32a6b1eb6a2ee449cd631468653f34af440bff66ec1474fb27873c66d421bda" Oct 31 22:17:26.035539 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:26.035512 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq"] Oct 31 22:17:26.047048 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:26.047022 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn"] Oct 31 22:17:26.049510 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:26.049493 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-277359d2-hpcfq"] Oct 31 22:17:26.051895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:26.051878 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2a4f08d-wglrn"] Oct 31 22:17:27.678516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:27.678478 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" containerID="cri-o://280d59643a65b156403084746f9a87368dc772271dbaa51ef5c3f078c7bd279a" gracePeriod=45 Oct 31 22:17:27.683665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:27.683635 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" containerID="cri-o://815e4ef4c4909ca26c35256c7818eebe42d5c3972b23622adbffe302638c49f2" gracePeriod=45 Oct 31 22:17:27.695790 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:27.695748 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" containerID="cri-o://08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178" gracePeriod=45 Oct 31 22:17:27.989642 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:27.989610 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="223b7c62-329f-4557-b2f1-5d5cef390870" path="/var/lib/kubelet/pods/223b7c62-329f-4557-b2f1-5d5cef390870/volumes" Oct 31 22:17:27.990142 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:27.990125 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82988d9b-07ca-4675-a72a-58b290f7cfcb" path="/var/lib/kubelet/pods/82988d9b-07ca-4675-a72a-58b290f7cfcb/volumes" Oct 31 22:17:28.002008 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.001987 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 22:17:28.015262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.015236 4294 generic.go:356] "Generic (PLEG): container finished" podID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerID="815e4ef4c4909ca26c35256c7818eebe42d5c3972b23622adbffe302638c49f2" exitCode=0 Oct 31 22:17:28.015555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.015335 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" event={"ID":"937f9e18-5cf6-4f15-b70e-7c960f57a58d","Type":"ContainerDied","Data":"815e4ef4c4909ca26c35256c7818eebe42d5c3972b23622adbffe302638c49f2"} Oct 31 22:17:28.017660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.017636 4294 generic.go:356] "Generic (PLEG): container finished" podID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerID="08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178" exitCode=0 Oct 31 22:17:28.017762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.017747 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" event={"ID":"2f2d31b6-3ec4-4fd2-8621-90b296663dc9","Type":"ContainerDied","Data":"08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178"} Oct 31 22:17:28.017840 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.017776 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" event={"ID":"2f2d31b6-3ec4-4fd2-8621-90b296663dc9","Type":"ContainerDied","Data":"1717b4278ac78f23cc50ba916220eed32f590ec9e42e6869b7900d7da2b2bbf3"} Oct 31 22:17:28.017840 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.017799 4294 scope.go:122] "RemoveContainer" containerID="08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178" Oct 31 22:17:28.017953 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.017933 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s" Oct 31 22:17:28.020989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.020959 4294 generic.go:356] "Generic (PLEG): container finished" podID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerID="280d59643a65b156403084746f9a87368dc772271dbaa51ef5c3f078c7bd279a" exitCode=0 Oct 31 22:17:28.021090 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.020993 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" event={"ID":"e322330f-18fb-405a-9af5-2fa33e9094c0","Type":"ContainerDied","Data":"280d59643a65b156403084746f9a87368dc772271dbaa51ef5c3f078c7bd279a"} Oct 31 22:17:28.021282 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.021262 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 22:17:28.030476 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.030459 4294 scope.go:122] "RemoveContainer" containerID="08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178" Oct 31 22:17:28.030708 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:17:28.030690 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178\": container with ID starting with 08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178 not found: ID does not exist" containerID="08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178" Oct 31 22:17:28.030786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.030719 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178"} err="failed to get container status \"08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178\": rpc error: code = NotFound desc = could not find container \"08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178\": container with ID starting with 08be3093bd1004d89675eb05fefcccc46db8a5f77f9670192d883a8752cb4178 not found: ID does not exist" Oct 31 22:17:28.032805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.032786 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 22:17:28.173603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173568 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tls-certs\" (UniqueName: \"kubernetes.io/secret/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tls-certs\") pod \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " Oct 31 22:17:28.173788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173625 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-tmp-dir\") pod \"e322330f-18fb-405a-9af5-2fa33e9094c0\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " Oct 31 22:17:28.173788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173675 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e322330f-18fb-405a-9af5-2fa33e9094c0-kube-api-access-58kzp\" (UniqueName: \"kubernetes.io/projected/e322330f-18fb-405a-9af5-2fa33e9094c0-kube-api-access-58kzp\") pod \"e322330f-18fb-405a-9af5-2fa33e9094c0\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " Oct 31 22:17:28.173788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173720 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937f9e18-5cf6-4f15-b70e-7c960f57a58d-kube-api-access-v4gkq\" (UniqueName: \"kubernetes.io/projected/937f9e18-5cf6-4f15-b70e-7c960f57a58d-kube-api-access-v4gkq\") pod \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " Oct 31 22:17:28.173788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173781 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-home\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-home\") pod \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " Oct 31 22:17:28.173946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173806 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-home\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-home\") pod \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " Oct 31 22:17:28.173946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173844 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tmp-dir\") pod \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " Oct 31 22:17:28.174032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.173953 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tls-certs\") pod \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " Oct 31 22:17:28.174032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174003 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-dshm\") pod \"e322330f-18fb-405a-9af5-2fa33e9094c0\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " Oct 31 22:17:28.174102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174046 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tmp-dir\") pod \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " Oct 31 22:17:28.174102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174087 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e322330f-18fb-405a-9af5-2fa33e9094c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/e322330f-18fb-405a-9af5-2fa33e9094c0-tls-certs\") pod \"e322330f-18fb-405a-9af5-2fa33e9094c0\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " Oct 31 22:17:28.174164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174117 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-dshm\") pod \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " Oct 31 22:17:28.174201 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174161 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-home\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-home\") pod \"e322330f-18fb-405a-9af5-2fa33e9094c0\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " Oct 31 22:17:28.174241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174231 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-model-cache\") pod \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " Oct 31 22:17:28.174338 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174316 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-kube-api-access-wvpnh\" (UniqueName: \"kubernetes.io/projected/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-kube-api-access-wvpnh\") pod \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " Oct 31 22:17:28.174396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174376 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-dshm\") pod \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\" (UID: \"937f9e18-5cf6-4f15-b70e-7c960f57a58d\") " Oct 31 22:17:28.174438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174420 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-home" pod "937f9e18-5cf6-4f15-b70e-7c960f57a58d" (UID: "937f9e18-5cf6-4f15-b70e-7c960f57a58d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.174438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174428 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-model-cache\") pod \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\" (UID: \"2f2d31b6-3ec4-4fd2-8621-90b296663dc9\") " Oct 31 22:17:28.174509 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.174458 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-model-cache\") pod \"e322330f-18fb-405a-9af5-2fa33e9094c0\" (UID: \"e322330f-18fb-405a-9af5-2fa33e9094c0\") " Oct 31 22:17:28.175110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.175075 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.175247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.175222 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tmp-dir" pod "2f2d31b6-3ec4-4fd2-8621-90b296663dc9" (UID: "2f2d31b6-3ec4-4fd2-8621-90b296663dc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.175414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.175247 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tmp-dir" pod "937f9e18-5cf6-4f15-b70e-7c960f57a58d" (UID: "937f9e18-5cf6-4f15-b70e-7c960f57a58d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.175504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.175349 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-model-cache" pod "e322330f-18fb-405a-9af5-2fa33e9094c0" (UID: "e322330f-18fb-405a-9af5-2fa33e9094c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.175807 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.175787 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-model-cache" pod "2f2d31b6-3ec4-4fd2-8621-90b296663dc9" (UID: "2f2d31b6-3ec4-4fd2-8621-90b296663dc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.178935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.178733 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tls-certs" pod "2f2d31b6-3ec4-4fd2-8621-90b296663dc9" (UID: "2f2d31b6-3ec4-4fd2-8621-90b296663dc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:28.179042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.178979 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-dshm" pod "2f2d31b6-3ec4-4fd2-8621-90b296663dc9" (UID: "2f2d31b6-3ec4-4fd2-8621-90b296663dc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.179462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179178 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-tmp-dir" pod "e322330f-18fb-405a-9af5-2fa33e9094c0" (UID: "e322330f-18fb-405a-9af5-2fa33e9094c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.179462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179222 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-home" pod "2f2d31b6-3ec4-4fd2-8621-90b296663dc9" (UID: "2f2d31b6-3ec4-4fd2-8621-90b296663dc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.179462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179253 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tls-certs" pod "937f9e18-5cf6-4f15-b70e-7c960f57a58d" (UID: "937f9e18-5cf6-4f15-b70e-7c960f57a58d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:28.179685 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179482 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-home" pod "e322330f-18fb-405a-9af5-2fa33e9094c0" (UID: "e322330f-18fb-405a-9af5-2fa33e9094c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.179754 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179713 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e322330f-18fb-405a-9af5-2fa33e9094c0-tls-certs" pod "e322330f-18fb-405a-9af5-2fa33e9094c0" (UID: "e322330f-18fb-405a-9af5-2fa33e9094c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:28.179818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179796 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-kube-api-access-wvpnh" pod "2f2d31b6-3ec4-4fd2-8621-90b296663dc9" (UID: "2f2d31b6-3ec4-4fd2-8621-90b296663dc9"). InnerVolumeSpecName "kube-api-access-wvpnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:28.179876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179831 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e322330f-18fb-405a-9af5-2fa33e9094c0-kube-api-access-58kzp" pod "e322330f-18fb-405a-9af5-2fa33e9094c0" (UID: "e322330f-18fb-405a-9af5-2fa33e9094c0"). InnerVolumeSpecName "kube-api-access-58kzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:28.180016 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.179988 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-model-cache" pod "937f9e18-5cf6-4f15-b70e-7c960f57a58d" (UID: "937f9e18-5cf6-4f15-b70e-7c960f57a58d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.180158 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.180138 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-dshm" pod "937f9e18-5cf6-4f15-b70e-7c960f57a58d" (UID: "937f9e18-5cf6-4f15-b70e-7c960f57a58d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.181218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.181194 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937f9e18-5cf6-4f15-b70e-7c960f57a58d-kube-api-access-v4gkq" pod "937f9e18-5cf6-4f15-b70e-7c960f57a58d" (UID: "937f9e18-5cf6-4f15-b70e-7c960f57a58d"). InnerVolumeSpecName "kube-api-access-v4gkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:28.181528 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.181505 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-dshm" pod "e322330f-18fb-405a-9af5-2fa33e9094c0" (UID: "e322330f-18fb-405a-9af5-2fa33e9094c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:28.245823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.245787 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" podUID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:17:28.245964 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.245823 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" podUID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:17:28.276722 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276698 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276722 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276719 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276744 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276752 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e322330f-18fb-405a-9af5-2fa33e9094c0-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276759 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276767 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276774 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276782 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvpnh\" (UniqueName: \"kubernetes.io/projected/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-kube-api-access-wvpnh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276790 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/937f9e18-5cf6-4f15-b70e-7c960f57a58d-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276797 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276805 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276813 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/937f9e18-5cf6-4f15-b70e-7c960f57a58d-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276821 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e322330f-18fb-405a-9af5-2fa33e9094c0-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276828 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58kzp\" (UniqueName: \"kubernetes.io/projected/e322330f-18fb-405a-9af5-2fa33e9094c0-kube-api-access-58kzp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276835 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4gkq\" (UniqueName: \"kubernetes.io/projected/937f9e18-5cf6-4f15-b70e-7c960f57a58d-kube-api-access-v4gkq\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276843 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.276864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.276850 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f2d31b6-3ec4-4fd2-8621-90b296663dc9-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:28.364411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.364381 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s"] Oct 31 22:17:28.368479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:28.368454 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/simulator-w2-c16684-kserve-5dc7686c49-vf79s"] Oct 31 22:17:29.030041 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.030014 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" Oct 31 22:17:29.030546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.030016 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls" event={"ID":"937f9e18-5cf6-4f15-b70e-7c960f57a58d","Type":"ContainerDied","Data":"7285f937c30863ae75680e9048989711ca84e781025b2ea204e03b4d958998b7"} Oct 31 22:17:29.030546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.030157 4294 scope.go:122] "RemoveContainer" containerID="815e4ef4c4909ca26c35256c7818eebe42d5c3972b23622adbffe302638c49f2" Oct 31 22:17:29.033020 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.032989 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" event={"ID":"e322330f-18fb-405a-9af5-2fa33e9094c0","Type":"ContainerDied","Data":"79524ebc858710a03371fcbc6038f17bd617ec6599bffc61eb9cc6675c62e110"} Oct 31 22:17:29.033125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.033003 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6" Oct 31 22:17:29.041735 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.041713 4294 scope.go:122] "RemoveContainer" containerID="280d59643a65b156403084746f9a87368dc772271dbaa51ef5c3f078c7bd279a" Oct 31 22:17:29.082645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.082615 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6"] Oct 31 22:17:29.088854 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.088832 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/distinct-w2-c16684-kserve-544c778c55-lcrw6"] Oct 31 22:17:29.121732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.121705 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls"] Oct 31 22:17:29.126098 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.126077 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c16684/premium-w2-c16684-kserve-6c48ffc6d8-8hxls"] Oct 31 22:17:29.996496 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.996460 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" path="/var/lib/kubelet/pods/2f2d31b6-3ec4-4fd2-8621-90b296663dc9/volumes" Oct 31 22:17:29.997103 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.997079 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" path="/var/lib/kubelet/pods/937f9e18-5cf6-4f15-b70e-7c960f57a58d/volumes" Oct 31 22:17:29.998144 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:29.998123 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" path="/var/lib/kubelet/pods/e322330f-18fb-405a-9af5-2fa33e9094c0/volumes" Oct 31 22:17:31.309938 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:17:31.309907 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:17:31.760727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:31.760692 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d"] Oct 31 22:17:31.760976 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:31.760944 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" containerID="cri-o://bc8fd27f1969e827304779455ca4241c0b70ea2723094dcb53bcdc9fd3a38971" gracePeriod=30 Oct 31 22:17:32.376403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.376370 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz"] Oct 31 22:17:32.377088 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377065 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" Oct 31 22:17:32.377181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377089 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" Oct 31 22:17:32.377181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377105 4294 cpu_manager.go:414] "containerMap: removing container" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" Oct 31 22:17:32.377181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377114 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" Oct 31 22:17:32.377181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377137 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" Oct 31 22:17:32.377181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377146 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" Oct 31 22:17:32.377181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377166 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ce717cad-91fe-441e-b3ad-f6fff5d888fa" containerName="revoke-keys" Oct 31 22:17:32.377181 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377174 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce717cad-91fe-441e-b3ad-f6fff5d888fa" containerName="revoke-keys" Oct 31 22:17:32.377568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377189 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0fcd8c4f-3298-4c2d-82a7-325ca715c9ac" containerName="revoke-keys" Oct 31 22:17:32.377568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377197 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fcd8c4f-3298-4c2d-82a7-325ca715c9ac" containerName="revoke-keys" Oct 31 22:17:32.377568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377353 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f2d31b6-3ec4-4fd2-8621-90b296663dc9" containerName="main" Oct 31 22:17:32.377568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377369 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="937f9e18-5cf6-4f15-b70e-7c960f57a58d" containerName="main" Oct 31 22:17:32.377568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377380 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e322330f-18fb-405a-9af5-2fa33e9094c0" containerName="main" Oct 31 22:17:32.377568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377391 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce717cad-91fe-441e-b3ad-f6fff5d888fa" containerName="revoke-keys" Oct 31 22:17:32.377568 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.377400 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fcd8c4f-3298-4c2d-82a7-325ca715c9ac" containerName="revoke-keys" Oct 31 22:17:32.382393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.382374 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.399334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.399282 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz"] Oct 31 22:17:32.426409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.426372 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffs4\" (UniqueName: \"kubernetes.io/projected/8cdbff46-1c28-4105-90f8-4b658b6555a4-kube-api-access-8ffs4\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.426555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.426438 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.426555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.426507 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.448931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.448911 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:17:32.449054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.448983 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:17:32.528019 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.527980 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.528171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.528029 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.528171 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.528149 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffs4\" (UniqueName: \"kubernetes.io/projected/8cdbff46-1c28-4105-90f8-4b658b6555a4-kube-api-access-8ffs4\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.528703 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.528679 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.530398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.530381 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.552208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.552175 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffs4\" (UniqueName: \"kubernetes.io/projected/8cdbff46-1c28-4105-90f8-4b658b6555a4-kube-api-access-8ffs4\") pod \"maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:32.697695 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:32.697612 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:33.003673 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:33.003635 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz"] Oct 31 22:17:33.005218 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:17:33.005182 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdbff46_1c28_4105_90f8_4b658b6555a4.slice/crio-ea0f417e88e6aa927a6f3bdf0334b68697442bb2c3c211d3259f3bad65d1c9d3 WatchSource:0}: Error finding container ea0f417e88e6aa927a6f3bdf0334b68697442bb2c3c211d3259f3bad65d1c9d3: Status 404 returned error can't find the container with id ea0f417e88e6aa927a6f3bdf0334b68697442bb2c3c211d3259f3bad65d1c9d3 Oct 31 22:17:33.061902 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:33.061871 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" event={"ID":"8cdbff46-1c28-4105-90f8-4b658b6555a4","Type":"ContainerStarted","Data":"ea0f417e88e6aa927a6f3bdf0334b68697442bb2c3c211d3259f3bad65d1c9d3"} Oct 31 22:17:33.492004 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:33.491970 4294 patch_prober.go:29] interesting pod/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:17:33.492366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:33.492020 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:17:34.070060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:34.069973 4294 generic.go:356] "Generic (PLEG): container finished" podID="8cdbff46-1c28-4105-90f8-4b658b6555a4" containerID="b4156dbbd9b096ad7fd9d7ba99204e88ce22cc6e453d823ff6af804d5995dfe2" exitCode=0 Oct 31 22:17:34.070221 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:34.070058 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" event={"ID":"8cdbff46-1c28-4105-90f8-4b658b6555a4","Type":"ContainerDied","Data":"b4156dbbd9b096ad7fd9d7ba99204e88ce22cc6e453d823ff6af804d5995dfe2"} Oct 31 22:17:35.226408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.226380 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:35.262156 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.262121 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-xks-ca\") pod \"8cdbff46-1c28-4105-90f8-4b658b6555a4\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " Oct 31 22:17:35.262369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.262268 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-openshift-ca\") pod \"8cdbff46-1c28-4105-90f8-4b658b6555a4\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " Oct 31 22:17:35.262441 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.262398 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cdbff46-1c28-4105-90f8-4b658b6555a4-kube-api-access-8ffs4\" (UniqueName: \"kubernetes.io/projected/8cdbff46-1c28-4105-90f8-4b658b6555a4-kube-api-access-8ffs4\") pod \"8cdbff46-1c28-4105-90f8-4b658b6555a4\" (UID: \"8cdbff46-1c28-4105-90f8-4b658b6555a4\") " Oct 31 22:17:35.262601 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.262579 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-openshift-ca" pod "8cdbff46-1c28-4105-90f8-4b658b6555a4" (UID: "8cdbff46-1c28-4105-90f8-4b658b6555a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:35.262778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.262754 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:35.264282 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.264256 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-xks-ca" pod "8cdbff46-1c28-4105-90f8-4b658b6555a4" (UID: "8cdbff46-1c28-4105-90f8-4b658b6555a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:35.264448 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.264287 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdbff46-1c28-4105-90f8-4b658b6555a4-kube-api-access-8ffs4" pod "8cdbff46-1c28-4105-90f8-4b658b6555a4" (UID: "8cdbff46-1c28-4105-90f8-4b658b6555a4"). InnerVolumeSpecName "kube-api-access-8ffs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:35.363489 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.363407 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cdbff46-1c28-4105-90f8-4b658b6555a4-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:35.363489 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:35.363436 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffs4\" (UniqueName: \"kubernetes.io/projected/8cdbff46-1c28-4105-90f8-4b658b6555a4-kube-api-access-8ffs4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:36.087593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:36.087554 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" Oct 31 22:17:36.087861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:36.087610 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz" event={"ID":"8cdbff46-1c28-4105-90f8-4b658b6555a4","Type":"ContainerDied","Data":"ea0f417e88e6aa927a6f3bdf0334b68697442bb2c3c211d3259f3bad65d1c9d3"} Oct 31 22:17:36.087861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:36.087658 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0f417e88e6aa927a6f3bdf0334b68697442bb2c3c211d3259f3bad65d1c9d3" Oct 31 22:17:36.671013 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:36.670978 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5"] Oct 31 22:17:36.671535 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:36.671238 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" podUID="7c46007a-1a41-45b4-b713-96d505118ec4" containerName="maas-api" containerID="cri-o://6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d" gracePeriod=30 Oct 31 22:17:37.050227 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.049902 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 22:17:37.133325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.113939 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c46007a-1a41-45b4-b713-96d505118ec4-kube-api-access-b4pbh\" (UniqueName: \"kubernetes.io/projected/7c46007a-1a41-45b4-b713-96d505118ec4-kube-api-access-b4pbh\") pod \"7c46007a-1a41-45b4-b713-96d505118ec4\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " Oct 31 22:17:37.133325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.114166 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c46007a-1a41-45b4-b713-96d505118ec4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c46007a-1a41-45b4-b713-96d505118ec4-openshift-service-ca\") pod \"7c46007a-1a41-45b4-b713-96d505118ec4\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " Oct 31 22:17:37.133325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.114342 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-maas-api-tls\") pod \"7c46007a-1a41-45b4-b713-96d505118ec4\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " Oct 31 22:17:37.133325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.114434 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-metrics-certs\") pod \"7c46007a-1a41-45b4-b713-96d505118ec4\" (UID: \"7c46007a-1a41-45b4-b713-96d505118ec4\") " Oct 31 22:17:37.133325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.129156 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c46007a-1a41-45b4-b713-96d505118ec4-openshift-service-ca" pod "7c46007a-1a41-45b4-b713-96d505118ec4" (UID: "7c46007a-1a41-45b4-b713-96d505118ec4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:37.169274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.158128 4294 generic.go:356] "Generic (PLEG): container finished" podID="7c46007a-1a41-45b4-b713-96d505118ec4" containerID="6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d" exitCode=0 Oct 31 22:17:37.169274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.158332 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" Oct 31 22:17:37.169274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.158771 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" event={"ID":"7c46007a-1a41-45b4-b713-96d505118ec4","Type":"ContainerDied","Data":"6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d"} Oct 31 22:17:37.169274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.158828 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5" event={"ID":"7c46007a-1a41-45b4-b713-96d505118ec4","Type":"ContainerDied","Data":"50aeb03a563b403153c20204182cfa16f13aa866af7fe0d32beb3360fb451fe9"} Oct 31 22:17:37.169274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.158860 4294 scope.go:122] "RemoveContainer" containerID="6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d" Oct 31 22:17:37.169274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.167208 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c46007a-1a41-45b4-b713-96d505118ec4-kube-api-access-b4pbh" pod "7c46007a-1a41-45b4-b713-96d505118ec4" (UID: "7c46007a-1a41-45b4-b713-96d505118ec4"). InnerVolumeSpecName "kube-api-access-b4pbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:37.169956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.169914 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-metrics-certs" pod "7c46007a-1a41-45b4-b713-96d505118ec4" (UID: "7c46007a-1a41-45b4-b713-96d505118ec4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:37.171395 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.170138 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-maas-api-tls" pod "7c46007a-1a41-45b4-b713-96d505118ec4" (UID: "7c46007a-1a41-45b4-b713-96d505118ec4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:37.174377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.173851 4294 generic.go:356] "Generic (PLEG): container finished" podID="4988780d-ff11-46b9-b92d-19f654a6d224" containerID="bc8fd27f1969e827304779455ca4241c0b70ea2723094dcb53bcdc9fd3a38971" exitCode=0 Oct 31 22:17:37.174377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.174030 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" event={"ID":"4988780d-ff11-46b9-b92d-19f654a6d224","Type":"ContainerDied","Data":"bc8fd27f1969e827304779455ca4241c0b70ea2723094dcb53bcdc9fd3a38971"} Oct 31 22:17:37.192485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.191076 4294 scope.go:122] "RemoveContainer" containerID="6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d" Oct 31 22:17:37.192485 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:17:37.191589 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d\": container with ID starting with 6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d not found: ID does not exist" containerID="6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d" Oct 31 22:17:37.192485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.191629 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d"} err="failed to get container status \"6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d\": rpc error: code = NotFound desc = could not find container \"6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d\": container with ID starting with 6863eee0e99d37f56d9b4fb9d7a606846b73992487f50c09f44855c1b73e624d not found: ID does not exist" Oct 31 22:17:37.216209 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.216182 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c46007a-1a41-45b4-b713-96d505118ec4-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.216209 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.216209 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.216434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.216220 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c46007a-1a41-45b4-b713-96d505118ec4-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.216434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.216230 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pbh\" (UniqueName: \"kubernetes.io/projected/7c46007a-1a41-45b4-b713-96d505118ec4-kube-api-access-b4pbh\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.229118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.229093 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:17:37.317291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317198 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-kube-api-access-7wht8\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-kube-api-access-7wht8\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317330 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-envoy\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317383 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-istio-token\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-istio-token\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317429 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-credential-socket\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317493 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317476 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-socket\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317503 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-data\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317633 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4988780d-ff11-46b9-b92d-19f654a6d224-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988780d-ff11-46b9-b92d-19f654a6d224-istio-podinfo\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317661 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-certs\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317719 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317694 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4988780d-ff11-46b9-b92d-19f654a6d224-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988780d-ff11-46b9-b92d-19f654a6d224-istiod-ca-cert\") pod \"4988780d-ff11-46b9-b92d-19f654a6d224\" (UID: \"4988780d-ff11-46b9-b92d-19f654a6d224\") " Oct 31 22:17:37.317930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317774 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-credential-socket" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:37.317930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.317826 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-data" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:37.318199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.318159 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4988780d-ff11-46b9-b92d-19f654a6d224-istiod-ca-cert" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:37.318359 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.318241 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-certs" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:37.318430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.318361 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.318430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.318385 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.318430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.318401 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4988780d-ff11-46b9-b92d-19f654a6d224-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.318613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.318591 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-socket" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:37.319753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.319726 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-istio-token" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:37.319940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.319915 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4988780d-ff11-46b9-b92d-19f654a6d224-istio-podinfo" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:17:37.319940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.319935 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-envoy" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:37.320063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.320022 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-kube-api-access-7wht8" pod "4988780d-ff11-46b9-b92d-19f654a6d224" (UID: "4988780d-ff11-46b9-b92d-19f654a6d224"). InnerVolumeSpecName "kube-api-access-7wht8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:37.402453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.402413 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w"] Oct 31 22:17:37.406038 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.406005 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-c34cbd-hmz4w"] Oct 31 22:17:37.419372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.419345 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.419372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.419371 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.419372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.419380 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.419595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.419389 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4988780d-ff11-46b9-b92d-19f654a6d224-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.419595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.419398 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4988780d-ff11-46b9-b92d-19f654a6d224-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.419595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.419407 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wht8\" (UniqueName: \"kubernetes.io/projected/4988780d-ff11-46b9-b92d-19f654a6d224-kube-api-access-7wht8\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:37.504785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.504749 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5"] Oct 31 22:17:37.509400 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.509375 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-c34cbd-6c78dfc577-pl4k5"] Oct 31 22:17:37.984374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.984347 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e89236-e66e-4aa6-81fa-ce6c4a920ff3" path="/var/lib/kubelet/pods/54e89236-e66e-4aa6-81fa-ce6c4a920ff3/volumes" Oct 31 22:17:37.984796 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:37.984782 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c46007a-1a41-45b4-b713-96d505118ec4" path="/var/lib/kubelet/pods/7c46007a-1a41-45b4-b713-96d505118ec4/volumes" Oct 31 22:17:38.183965 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:38.183938 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" Oct 31 22:17:38.183965 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:38.183956 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d" event={"ID":"4988780d-ff11-46b9-b92d-19f654a6d224","Type":"ContainerDied","Data":"c74c85c81e48aacf2306d7192a26ad420eca7d36d18113608c1e4e2cfe8131ae"} Oct 31 22:17:38.184240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:38.184017 4294 scope.go:122] "RemoveContainer" containerID="bc8fd27f1969e827304779455ca4241c0b70ea2723094dcb53bcdc9fd3a38971" Oct 31 22:17:38.239458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:38.239388 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d"] Oct 31 22:17:38.246555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:38.246526 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" podUID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:17:38.246555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:38.246542 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" podUID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:17:38.251218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:38.251191 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-cd421c-openshift-default-5b7f98b7b5-ltb5d"] Oct 31 22:17:39.195765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.195735 4294 generic.go:356] "Generic (PLEG): container finished" podID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" containerID="6ebcbf6ce44f0ba414255b80a4765ba0fe169ced5318e75ecec533d23dbd700f" exitCode=0 Oct 31 22:17:39.195765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.195758 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" event={"ID":"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4","Type":"ContainerDied","Data":"6ebcbf6ce44f0ba414255b80a4765ba0fe169ced5318e75ecec533d23dbd700f"} Oct 31 22:17:39.197595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.197572 4294 generic.go:356] "Generic (PLEG): container finished" podID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" containerID="86151cacf887b641f5139e4eaaa8690e0a6fc291b1959205f267e212417220d5" exitCode=0 Oct 31 22:17:39.197720 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.197642 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" event={"ID":"ca5b6cba-bd54-4c8f-a351-26b092e20e20","Type":"ContainerDied","Data":"86151cacf887b641f5139e4eaaa8690e0a6fc291b1959205f267e212417220d5"} Oct 31 22:17:39.329278 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.329254 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:17:39.381705 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.381679 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:17:39.443017 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.442976 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-plugins-config-volume\") pod \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " Oct 31 22:17:39.443224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.443047 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5b6cba-bd54-4c8f-a351-26b092e20e20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5b6cba-bd54-4c8f-a351-26b092e20e20-plugins-config-volume\") pod \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " Oct 31 22:17:39.443224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.443185 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5b6cba-bd54-4c8f-a351-26b092e20e20-kube-api-access-qsjrv\" (UniqueName: \"kubernetes.io/projected/ca5b6cba-bd54-4c8f-a351-26b092e20e20-kube-api-access-qsjrv\") pod \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\" (UID: \"ca5b6cba-bd54-4c8f-a351-26b092e20e20\") " Oct 31 22:17:39.443224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.443213 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-kube-api-access-zhzhj\" (UniqueName: \"kubernetes.io/projected/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-kube-api-access-zhzhj\") pod \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\" (UID: \"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4\") " Oct 31 22:17:39.443455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.443373 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-plugins-config-volume" pod "80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" (UID: "80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:39.443519 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.443455 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5b6cba-bd54-4c8f-a351-26b092e20e20-plugins-config-volume" pod "ca5b6cba-bd54-4c8f-a351-26b092e20e20" (UID: "ca5b6cba-bd54-4c8f-a351-26b092e20e20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:39.443784 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.443758 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca5b6cba-bd54-4c8f-a351-26b092e20e20-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:39.443864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.443788 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:39.445426 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.445402 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5b6cba-bd54-4c8f-a351-26b092e20e20-kube-api-access-qsjrv" pod "ca5b6cba-bd54-4c8f-a351-26b092e20e20" (UID: "ca5b6cba-bd54-4c8f-a351-26b092e20e20"). InnerVolumeSpecName "kube-api-access-qsjrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:39.445783 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.445760 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-kube-api-access-zhzhj" pod "80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" (UID: "80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4"). InnerVolumeSpecName "kube-api-access-zhzhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:39.544954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.544876 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsjrv\" (UniqueName: \"kubernetes.io/projected/ca5b6cba-bd54-4c8f-a351-26b092e20e20-kube-api-access-qsjrv\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:39.544954 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.544905 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhzhj\" (UniqueName: \"kubernetes.io/projected/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4-kube-api-access-zhzhj\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:39.984865 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:39.984832 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" path="/var/lib/kubelet/pods/4988780d-ff11-46b9-b92d-19f654a6d224/volumes" Oct 31 22:17:40.206530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.206493 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" Oct 31 22:17:40.206530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.206508 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s" event={"ID":"80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4","Type":"ContainerDied","Data":"b09b7f4d34f912e890bde2372cce48209e53495b6ffb2782dfeb803847c97456"} Oct 31 22:17:40.207101 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.206573 4294 scope.go:122] "RemoveContainer" containerID="6ebcbf6ce44f0ba414255b80a4765ba0fe169ced5318e75ecec533d23dbd700f" Oct 31 22:17:40.208042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.207963 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" Oct 31 22:17:40.208042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.207988 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6" event={"ID":"ca5b6cba-bd54-4c8f-a351-26b092e20e20","Type":"ContainerDied","Data":"870499022b5c4c01f8cc6a6daaf55fedb4ee55355715097cc7460e2dca81ab64"} Oct 31 22:17:40.216940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.216921 4294 scope.go:122] "RemoveContainer" containerID="86151cacf887b641f5139e4eaaa8690e0a6fc291b1959205f267e212417220d5" Oct 31 22:17:40.260379 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.260286 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6"] Oct 31 22:17:40.275979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.275929 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-cd421c-6884df5c6c-ns6q6"] Oct 31 22:17:40.297753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.297721 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s"] Oct 31 22:17:40.304348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:40.304320 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-cd421c-5f4c88c68c-8hf7s"] Oct 31 22:17:41.984505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:41.984473 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" path="/var/lib/kubelet/pods/80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4/volumes" Oct 31 22:17:42.012645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:41.984899 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" path="/var/lib/kubelet/pods/ca5b6cba-bd54-4c8f-a351-26b092e20e20/volumes" Oct 31 22:17:46.156497 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.156441 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb"] Oct 31 22:17:46.157628 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.157595 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" containerID="cri-o://8f00abbafeb515544bff6908092eba8d8cfce091c98e8b570c1435ed71087bd1" gracePeriod=30 Oct 31 22:17:46.739529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.739490 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw"] Oct 31 22:17:46.740583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.740554 4294 cpu_manager.go:414] "containerMap: removing container" podUID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" containerName="payload-pre-processing" Oct 31 22:17:46.740751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.740737 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" containerName="payload-pre-processing" Oct 31 22:17:46.740850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.740840 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" Oct 31 22:17:46.740926 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.740916 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" Oct 31 22:17:46.741028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741018 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8cdbff46-1c28-4105-90f8-4b658b6555a4" containerName="revoke-keys" Oct 31 22:17:46.741104 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741094 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cdbff46-1c28-4105-90f8-4b658b6555a4" containerName="revoke-keys" Oct 31 22:17:46.741224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741211 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" containerName="payload-processing" Oct 31 22:17:46.741326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741315 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" containerName="payload-processing" Oct 31 22:17:46.741445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741436 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7c46007a-1a41-45b4-b713-96d505118ec4" containerName="maas-api" Oct 31 22:17:46.741523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741513 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c46007a-1a41-45b4-b713-96d505118ec4" containerName="maas-api" Oct 31 22:17:46.741740 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741728 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c46007a-1a41-45b4-b713-96d505118ec4" containerName="maas-api" Oct 31 22:17:46.741833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741823 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c77e2f-514a-4ddf-9f8e-3ec45d1fdbf4" containerName="payload-pre-processing" Oct 31 22:17:46.741919 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741908 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4988780d-ff11-46b9-b92d-19f654a6d224" containerName="istio-proxy" Oct 31 22:17:46.742001 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.741992 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cdbff46-1c28-4105-90f8-4b658b6555a4" containerName="revoke-keys" Oct 31 22:17:46.742090 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.742081 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5b6cba-bd54-4c8f-a351-26b092e20e20" containerName="payload-processing" Oct 31 22:17:46.746163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.746140 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.755216 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.755085 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw"] Oct 31 22:17:46.816500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.816395 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvpb\" (UniqueName: \"kubernetes.io/projected/89bae720-5e95-4a31-8e6d-758190101956-kube-api-access-dbvpb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.852686 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.852647 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:17:46.852901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.852811 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:17:46.861587 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.816960 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bae720-5e95-4a31-8e6d-758190101956-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.861587 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.859509 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bae720-5e95-4a31-8e6d-758190101956-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.960692 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.960653 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bae720-5e95-4a31-8e6d-758190101956-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.960888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.960735 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bae720-5e95-4a31-8e6d-758190101956-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.960888 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.960795 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvpb\" (UniqueName: \"kubernetes.io/projected/89bae720-5e95-4a31-8e6d-758190101956-kube-api-access-dbvpb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.961550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.961497 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bae720-5e95-4a31-8e6d-758190101956-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.963648 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.963622 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bae720-5e95-4a31-8e6d-758190101956-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:46.984203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:46.984168 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvpb\" (UniqueName: \"kubernetes.io/projected/89bae720-5e95-4a31-8e6d-758190101956-kube-api-access-dbvpb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:47.104165 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:47.104059 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:47.431042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:47.431006 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw"] Oct 31 22:17:47.432545 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:17:47.432512 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bae720_5e95_4a31_8e6d_758190101956.slice/crio-ad8e4abc348af22a334584ef833fde3e6ad9774c7a5ae6a8304946304912c9d6 WatchSource:0}: Error finding container ad8e4abc348af22a334584ef833fde3e6ad9774c7a5ae6a8304946304912c9d6: Status 404 returned error can't find the container with id ad8e4abc348af22a334584ef833fde3e6ad9774c7a5ae6a8304946304912c9d6 Oct 31 22:17:48.365377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:48.365342 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" event={"ID":"89bae720-5e95-4a31-8e6d-758190101956","Type":"ContainerStarted","Data":"80f59048cce631d346c6be7c9237b1f33f0f6b35b8e9706a86622ff9f7c05a56"} Oct 31 22:17:48.365567 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:48.365409 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" event={"ID":"89bae720-5e95-4a31-8e6d-758190101956","Type":"ContainerStarted","Data":"ad8e4abc348af22a334584ef833fde3e6ad9774c7a5ae6a8304946304912c9d6"} Oct 31 22:17:48.405012 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:48.404955 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" podStartSLOduration=2.404937944 podStartE2EDuration="2.404937944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:17:48.404770681 +0000 UTC m=+3708.529877912" watchObservedRunningTime="2026-10-31 22:17:48.404937944 +0000 UTC m=+3708.530045182" Oct 31 22:17:49.373004 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:49.372973 4294 generic.go:356] "Generic (PLEG): container finished" podID="89bae720-5e95-4a31-8e6d-758190101956" containerID="80f59048cce631d346c6be7c9237b1f33f0f6b35b8e9706a86622ff9f7c05a56" exitCode=0 Oct 31 22:17:49.373494 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:49.373063 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" event={"ID":"89bae720-5e95-4a31-8e6d-758190101956","Type":"ContainerDied","Data":"80f59048cce631d346c6be7c9237b1f33f0f6b35b8e9706a86622ff9f7c05a56"} Oct 31 22:17:50.046547 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.046505 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b"] Oct 31 22:17:50.046757 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.046730 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" podUID="83a0a5c3-a26f-45b1-853f-242b1896dcff" containerName="maas-api" containerID="cri-o://82896a00941302883a2b1c3ae91ffd3cb46d25cfdb756364dc5be85a9fa0cc80" gracePeriod=30 Oct 31 22:17:50.382362 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.382327 4294 generic.go:356] "Generic (PLEG): container finished" podID="83a0a5c3-a26f-45b1-853f-242b1896dcff" containerID="82896a00941302883a2b1c3ae91ffd3cb46d25cfdb756364dc5be85a9fa0cc80" exitCode=0 Oct 31 22:17:50.382794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.382422 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" event={"ID":"83a0a5c3-a26f-45b1-853f-242b1896dcff","Type":"ContainerDied","Data":"82896a00941302883a2b1c3ae91ffd3cb46d25cfdb756364dc5be85a9fa0cc80"} Oct 31 22:17:50.382794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.382487 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" event={"ID":"83a0a5c3-a26f-45b1-853f-242b1896dcff","Type":"ContainerDied","Data":"34a788fb7753e08aa5516d54fa1075013850c9e4ef7a3421dda958ac1e08a2ea"} Oct 31 22:17:50.382794 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.382512 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34a788fb7753e08aa5516d54fa1075013850c9e4ef7a3421dda958ac1e08a2ea" Oct 31 22:17:50.385398 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.385371 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 22:17:50.530084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.528050 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-maas-api-tls\") pod \"83a0a5c3-a26f-45b1-853f-242b1896dcff\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " Oct 31 22:17:50.530084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.528172 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a0a5c3-a26f-45b1-853f-242b1896dcff-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a0a5c3-a26f-45b1-853f-242b1896dcff-openshift-service-ca\") pod \"83a0a5c3-a26f-45b1-853f-242b1896dcff\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " Oct 31 22:17:50.530084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.528408 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-metrics-certs\") pod \"83a0a5c3-a26f-45b1-853f-242b1896dcff\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " Oct 31 22:17:50.530084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.528464 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a0a5c3-a26f-45b1-853f-242b1896dcff-kube-api-access-kwqns\" (UniqueName: \"kubernetes.io/projected/83a0a5c3-a26f-45b1-853f-242b1896dcff-kube-api-access-kwqns\") pod \"83a0a5c3-a26f-45b1-853f-242b1896dcff\" (UID: \"83a0a5c3-a26f-45b1-853f-242b1896dcff\") " Oct 31 22:17:50.530084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.528926 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a0a5c3-a26f-45b1-853f-242b1896dcff-openshift-service-ca" pod "83a0a5c3-a26f-45b1-853f-242b1896dcff" (UID: "83a0a5c3-a26f-45b1-853f-242b1896dcff"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:50.530084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.529163 4294 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83a0a5c3-a26f-45b1-853f-242b1896dcff-openshift-service-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:50.556531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.547701 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-maas-api-tls" pod "83a0a5c3-a26f-45b1-853f-242b1896dcff" (UID: "83a0a5c3-a26f-45b1-853f-242b1896dcff"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:50.556531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.547743 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a0a5c3-a26f-45b1-853f-242b1896dcff-kube-api-access-kwqns" pod "83a0a5c3-a26f-45b1-853f-242b1896dcff" (UID: "83a0a5c3-a26f-45b1-853f-242b1896dcff"). InnerVolumeSpecName "kube-api-access-kwqns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:50.556531 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.547790 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-metrics-certs" pod "83a0a5c3-a26f-45b1-853f-242b1896dcff" (UID: "83a0a5c3-a26f-45b1-853f-242b1896dcff"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:50.574741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.574719 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:50.630402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.630340 4294 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-metrics-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:50.630402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.630365 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwqns\" (UniqueName: \"kubernetes.io/projected/83a0a5c3-a26f-45b1-853f-242b1896dcff-kube-api-access-kwqns\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:50.630402 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.630374 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83a0a5c3-a26f-45b1-853f-242b1896dcff-maas-api-tls\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:50.731498 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.731463 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89bae720-5e95-4a31-8e6d-758190101956-kube-api-access-dbvpb\" (UniqueName: \"kubernetes.io/projected/89bae720-5e95-4a31-8e6d-758190101956-kube-api-access-dbvpb\") pod \"89bae720-5e95-4a31-8e6d-758190101956\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " Oct 31 22:17:50.731669 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.731522 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89bae720-5e95-4a31-8e6d-758190101956-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bae720-5e95-4a31-8e6d-758190101956-maas-api-xks-ca\") pod \"89bae720-5e95-4a31-8e6d-758190101956\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " Oct 31 22:17:50.731763 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.731710 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89bae720-5e95-4a31-8e6d-758190101956-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bae720-5e95-4a31-8e6d-758190101956-maas-api-openshift-ca\") pod \"89bae720-5e95-4a31-8e6d-758190101956\" (UID: \"89bae720-5e95-4a31-8e6d-758190101956\") " Oct 31 22:17:50.732407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.732372 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89bae720-5e95-4a31-8e6d-758190101956-maas-api-openshift-ca" pod "89bae720-5e95-4a31-8e6d-758190101956" (UID: "89bae720-5e95-4a31-8e6d-758190101956"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:50.733712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.733681 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89bae720-5e95-4a31-8e6d-758190101956-kube-api-access-dbvpb" pod "89bae720-5e95-4a31-8e6d-758190101956" (UID: "89bae720-5e95-4a31-8e6d-758190101956"). InnerVolumeSpecName "kube-api-access-dbvpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:50.733833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.733784 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89bae720-5e95-4a31-8e6d-758190101956-maas-api-xks-ca" pod "89bae720-5e95-4a31-8e6d-758190101956" (UID: "89bae720-5e95-4a31-8e6d-758190101956"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:17:50.777936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.777906 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45"] Oct 31 22:17:50.780598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.780572 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c16684-qnm45"] Oct 31 22:17:50.833089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.833060 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvpb\" (UniqueName: \"kubernetes.io/projected/89bae720-5e95-4a31-8e6d-758190101956-kube-api-access-dbvpb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:50.833089 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.833086 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89bae720-5e95-4a31-8e6d-758190101956-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:50.833256 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:50.833098 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89bae720-5e95-4a31-8e6d-758190101956-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.394672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.394634 4294 generic.go:356] "Generic (PLEG): container finished" podID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerID="8f00abbafeb515544bff6908092eba8d8cfce091c98e8b570c1435ed71087bd1" exitCode=0 Oct 31 22:17:51.395118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.394698 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" event={"ID":"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab","Type":"ContainerDied","Data":"8f00abbafeb515544bff6908092eba8d8cfce091c98e8b570c1435ed71087bd1"} Oct 31 22:17:51.395118 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.394774 4294 scope.go:122] "RemoveContainer" containerID="4309e06a403166918045e0265f431e08c7791ecf4c98f85bad798b8d1846c8ba" Oct 31 22:17:51.396834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.396813 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" Oct 31 22:17:51.396834 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.396825 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw" event={"ID":"89bae720-5e95-4a31-8e6d-758190101956","Type":"ContainerDied","Data":"ad8e4abc348af22a334584ef833fde3e6ad9774c7a5ae6a8304946304912c9d6"} Oct 31 22:17:51.397033 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.396882 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad8e4abc348af22a334584ef833fde3e6ad9774c7a5ae6a8304946304912c9d6" Oct 31 22:17:51.397097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.397049 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b" Oct 31 22:17:51.453354 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.452039 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b"] Oct 31 22:17:51.456323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.455352 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 22:17:51.462365 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.462338 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c16684-c47f56d6b-rv29b"] Oct 31 22:17:51.543629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543602 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-certs\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.543787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543647 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-data\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.543787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543665 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-credential-socket\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.543787 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543762 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-socket\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.543959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543836 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-kube-api-access-j8pp4\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-kube-api-access-j8pp4\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.543959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543884 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-podinfo\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.543959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543906 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-certs" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:51.543959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543929 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-data" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:51.544163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.543994 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istiod-ca-cert\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.544163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544037 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-token\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-token\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.544163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544068 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-credential-socket" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:51.544163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544079 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-socket" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:51.544163 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544098 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-envoy\") pod \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\" (UID: \"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab\") " Oct 31 22:17:51.544586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544563 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istiod-ca-cert" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:17:51.544586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544582 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.544746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544600 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.544746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544612 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.544746 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.544622 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.546894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.546197 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-podinfo" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:17:51.546894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.546433 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-kube-api-access-j8pp4" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "kube-api-access-j8pp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:51.546894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.546542 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-token" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:17:51.547174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.547149 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-envoy" pod "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" (UID: "4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:17:51.645804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.645708 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8pp4\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-kube-api-access-j8pp4\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.645804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.645743 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.645804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.645757 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.645804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.645769 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.645804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.645783 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:17:51.984516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.984486 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="406c2372-7b78-422b-80bb-52622c8c23cf" path="/var/lib/kubelet/pods/406c2372-7b78-422b-80bb-52622c8c23cf/volumes" Oct 31 22:17:51.984952 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:51.984937 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a0a5c3-a26f-45b1-853f-242b1896dcff" path="/var/lib/kubelet/pods/83a0a5c3-a26f-45b1-853f-242b1896dcff/volumes" Oct 31 22:17:52.405187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:52.405101 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" Oct 31 22:17:52.405187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:52.405144 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb" event={"ID":"4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab","Type":"ContainerDied","Data":"a64559a8c370ee08b88c2f28549d290676f86b58220dad2cce35c9e75765e333"} Oct 31 22:17:52.405750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:52.405200 4294 scope.go:122] "RemoveContainer" containerID="8f00abbafeb515544bff6908092eba8d8cfce091c98e8b570c1435ed71087bd1" Oct 31 22:17:52.469397 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:52.469368 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb"] Oct 31 22:17:52.481487 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:52.481462 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-c34cbd-openshift-default-76b9b8b765-dp7tb"] Oct 31 22:17:53.984413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:53.984380 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" path="/var/lib/kubelet/pods/4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab/volumes" Oct 31 22:17:57.181585 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:57.181553 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf"] Oct 31 22:17:57.218559 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:57.218531 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw"] Oct 31 22:17:58.333530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:58.333494 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz"] Oct 31 22:17:58.344908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:58.344884 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c08360-vc9fz"] Oct 31 22:17:59.985548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:17:59.985511 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cdbff46-1c28-4105-90f8-4b658b6555a4" path="/var/lib/kubelet/pods/8cdbff46-1c28-4105-90f8-4b658b6555a4/volumes" Oct 31 22:18:01.310576 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:01.310516 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:18:02.182455 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:02.182416 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" podUID="b3972990-9501-4a33-97b9-84df70dcd649" containerName="payload-processing" containerID="cri-o://3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca" gracePeriod=25 Oct 31 22:18:02.219350 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:02.219286 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" podUID="04ac6d67-3ecb-4c43-9df3-b2a270278829" containerName="payload-pre-processing" containerID="cri-o://beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c" gracePeriod=25 Oct 31 22:18:04.099320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:04.099258 4294 scope.go:122] "RemoveContainer" containerID="b6360e12f197b0b857dfa72d69fcad312b3e6ef9baa2063bcfba5078cee9ceca" Oct 31 22:18:04.109251 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:04.109228 4294 scope.go:122] "RemoveContainer" containerID="82896a00941302883a2b1c3ae91ffd3cb46d25cfdb756364dc5be85a9fa0cc80" Oct 31 22:18:04.120047 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:04.120023 4294 scope.go:122] "RemoveContainer" containerID="b9157412e999d1b850ed433be43b78586cdd7398937958b1725d9d47db851807" Oct 31 22:18:06.043066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.043021 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z"] Oct 31 22:18:06.043598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.043447 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" containerID="cri-o://1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce" gracePeriod=30 Oct 31 22:18:06.567584 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.567551 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7"] Oct 31 22:18:06.568003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.567992 4294 cpu_manager.go:414] "containerMap: removing container" podUID="83a0a5c3-a26f-45b1-853f-242b1896dcff" containerName="maas-api" Oct 31 22:18:06.568044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568005 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a0a5c3-a26f-45b1-853f-242b1896dcff" containerName="maas-api" Oct 31 22:18:06.568044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568026 4294 cpu_manager.go:414] "containerMap: removing container" podUID="89bae720-5e95-4a31-8e6d-758190101956" containerName="revoke-keys" Oct 31 22:18:06.568044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568031 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89bae720-5e95-4a31-8e6d-758190101956" containerName="revoke-keys" Oct 31 22:18:06.568044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568043 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" Oct 31 22:18:06.568186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568048 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" Oct 31 22:18:06.568186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568071 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" Oct 31 22:18:06.568186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568076 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" Oct 31 22:18:06.568186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568140 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" Oct 31 22:18:06.568186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568147 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a0a5c3-a26f-45b1-853f-242b1896dcff" containerName="maas-api" Oct 31 22:18:06.568186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568154 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="89bae720-5e95-4a31-8e6d-758190101956" containerName="revoke-keys" Oct 31 22:18:06.568186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.568161 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba9dfb2-596f-4ade-85eb-ca423cbbd1ab" containerName="istio-proxy" Oct 31 22:18:06.571342 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.571325 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.590270 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.590242 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7"] Oct 31 22:18:06.609208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.609172 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbb5h\" (UniqueName: \"kubernetes.io/projected/900ff827-7636-42d2-9a5c-49c7a470e1a5-kube-api-access-lbb5h\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.609369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.609249 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.609509 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.609480 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.674308 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.674287 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:18:06.674450 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.674435 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:18:06.711105 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.711071 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbb5h\" (UniqueName: \"kubernetes.io/projected/900ff827-7636-42d2-9a5c-49c7a470e1a5-kube-api-access-lbb5h\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.711208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.711145 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.711268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.711223 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.711827 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.711800 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.713505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.713483 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.734204 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.734180 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbb5h\" (UniqueName: \"kubernetes.io/projected/900ff827-7636-42d2-9a5c-49c7a470e1a5-kube-api-access-lbb5h\") pod \"maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:06.883751 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:06.883699 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:07.213259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:07.211250 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7"] Oct 31 22:18:07.213259 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:18:07.212724 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900ff827_7636_42d2_9a5c_49c7a470e1a5.slice/crio-41db2d06b07fb6e1e74a88cc0548f04e1c26271b0b520f00312408b1635c32c3 WatchSource:0}: Error finding container 41db2d06b07fb6e1e74a88cc0548f04e1c26271b0b520f00312408b1635c32c3: Status 404 returned error can't find the container with id 41db2d06b07fb6e1e74a88cc0548f04e1c26271b0b520f00312408b1635c32c3 Oct 31 22:18:07.506990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:07.506961 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" event={"ID":"900ff827-7636-42d2-9a5c-49c7a470e1a5","Type":"ContainerStarted","Data":"e85c5b323d2729a484c02bae482746f04fd7570cda82538fe5a1f7606b9845d4"} Oct 31 22:18:07.507117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:07.507009 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" event={"ID":"900ff827-7636-42d2-9a5c-49c7a470e1a5","Type":"ContainerStarted","Data":"41db2d06b07fb6e1e74a88cc0548f04e1c26271b0b520f00312408b1635c32c3"} Oct 31 22:18:07.547594 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:07.547551 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" podStartSLOduration=1.547538069 podStartE2EDuration="1.547538069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:18:07.546761988 +0000 UTC m=+3727.671869219" watchObservedRunningTime="2026-10-31 22:18:07.547538069 +0000 UTC m=+3727.672645300" Oct 31 22:18:08.294650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.294621 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl"] Oct 31 22:18:08.298434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.298414 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.316767 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.316743 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl"] Oct 31 22:18:08.330238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.330144 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.330384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.330261 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbfml\" (UniqueName: \"kubernetes.io/projected/1158c05b-ae26-4736-b2fd-a632bb7fdf65-kube-api-access-xbfml\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.330384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.330347 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.431223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.431196 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.431414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.431252 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbfml\" (UniqueName: \"kubernetes.io/projected/1158c05b-ae26-4736-b2fd-a632bb7fdf65-kube-api-access-xbfml\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.431414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.431334 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.431961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.431942 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.433499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.433482 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.454576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.454548 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbfml\" (UniqueName: \"kubernetes.io/projected/1158c05b-ae26-4736-b2fd-a632bb7fdf65-kube-api-access-xbfml\") pod \"maas-api-revoke-sub-2869d921-n9lhl\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.514133 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.514099 4294 generic.go:356] "Generic (PLEG): container finished" podID="900ff827-7636-42d2-9a5c-49c7a470e1a5" containerID="e85c5b323d2729a484c02bae482746f04fd7570cda82538fe5a1f7606b9845d4" exitCode=0 Oct 31 22:18:08.514281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.514181 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" event={"ID":"900ff827-7636-42d2-9a5c-49c7a470e1a5","Type":"ContainerDied","Data":"e85c5b323d2729a484c02bae482746f04fd7570cda82538fe5a1f7606b9845d4"} Oct 31 22:18:08.612895 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.612809 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:08.916684 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:08.916647 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl"] Oct 31 22:18:08.917864 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:18:08.917838 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1158c05b_ae26_4736_b2fd_a632bb7fdf65.slice/crio-cd6e1d37f93268a551c1811568de47b612abdd7de0cd874067eba90b2a4bc515 WatchSource:0}: Error finding container cd6e1d37f93268a551c1811568de47b612abdd7de0cd874067eba90b2a4bc515: Status 404 returned error can't find the container with id cd6e1d37f93268a551c1811568de47b612abdd7de0cd874067eba90b2a4bc515 Oct 31 22:18:09.526393 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.526351 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" event={"ID":"1158c05b-ae26-4736-b2fd-a632bb7fdf65","Type":"ContainerStarted","Data":"f59fd1e28b7bb5f12acecf9abd56856498bb82c5b5c141e96cf34787a68542a7"} Oct 31 22:18:09.526793 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.526430 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" event={"ID":"1158c05b-ae26-4736-b2fd-a632bb7fdf65","Type":"ContainerStarted","Data":"cd6e1d37f93268a551c1811568de47b612abdd7de0cd874067eba90b2a4bc515"} Oct 31 22:18:09.541977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.541941 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" podUID="04ac6d67-3ecb-4c43-9df3-b2a270278829" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:09.542136 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.541964 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" podUID="b3972990-9501-4a33-97b9-84df70dcd649" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:09.572403 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.572355 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" podStartSLOduration=1.5723434840000001 podStartE2EDuration="1.572343484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:18:09.571872243 +0000 UTC m=+3729.696979474" watchObservedRunningTime="2026-10-31 22:18:09.572343484 +0000 UTC m=+3729.697450712" Oct 31 22:18:09.682221 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.682197 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:09.744900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.744873 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-openshift-ca\") pod \"900ff827-7636-42d2-9a5c-49c7a470e1a5\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " Oct 31 22:18:09.745057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.744917 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-xks-ca\") pod \"900ff827-7636-42d2-9a5c-49c7a470e1a5\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " Oct 31 22:18:09.745057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.744974 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900ff827-7636-42d2-9a5c-49c7a470e1a5-kube-api-access-lbb5h\" (UniqueName: \"kubernetes.io/projected/900ff827-7636-42d2-9a5c-49c7a470e1a5-kube-api-access-lbb5h\") pod \"900ff827-7636-42d2-9a5c-49c7a470e1a5\" (UID: \"900ff827-7636-42d2-9a5c-49c7a470e1a5\") " Oct 31 22:18:09.745275 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.745249 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-openshift-ca" pod "900ff827-7636-42d2-9a5c-49c7a470e1a5" (UID: "900ff827-7636-42d2-9a5c-49c7a470e1a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:09.745530 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.745505 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:09.746945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.746915 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-xks-ca" pod "900ff827-7636-42d2-9a5c-49c7a470e1a5" (UID: "900ff827-7636-42d2-9a5c-49c7a470e1a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:09.747028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.747007 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900ff827-7636-42d2-9a5c-49c7a470e1a5-kube-api-access-lbb5h" pod "900ff827-7636-42d2-9a5c-49c7a470e1a5" (UID: "900ff827-7636-42d2-9a5c-49c7a470e1a5"). InnerVolumeSpecName "kube-api-access-lbb5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:09.847091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.847037 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/900ff827-7636-42d2-9a5c-49c7a470e1a5-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:09.847091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:09.847061 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbb5h\" (UniqueName: \"kubernetes.io/projected/900ff827-7636-42d2-9a5c-49c7a470e1a5-kube-api-access-lbb5h\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:10.383733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:10.383698 4294 patch_prober.go:29] interesting pod/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:18:10.383896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:10.383752 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:18:10.534683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:10.534652 4294 generic.go:356] "Generic (PLEG): container finished" podID="1158c05b-ae26-4736-b2fd-a632bb7fdf65" containerID="f59fd1e28b7bb5f12acecf9abd56856498bb82c5b5c141e96cf34787a68542a7" exitCode=0 Oct 31 22:18:10.535162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:10.534736 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" event={"ID":"1158c05b-ae26-4736-b2fd-a632bb7fdf65","Type":"ContainerDied","Data":"f59fd1e28b7bb5f12acecf9abd56856498bb82c5b5c141e96cf34787a68542a7"} Oct 31 22:18:10.536396 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:10.536377 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" Oct 31 22:18:10.536504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:10.536401 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7" event={"ID":"900ff827-7636-42d2-9a5c-49c7a470e1a5","Type":"ContainerDied","Data":"41db2d06b07fb6e1e74a88cc0548f04e1c26271b0b520f00312408b1635c32c3"} Oct 31 22:18:10.536504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:10.536436 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41db2d06b07fb6e1e74a88cc0548f04e1c26271b0b520f00312408b1635c32c3" Oct 31 22:18:11.303060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.303038 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:18:11.362723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362654 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-token\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.362723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362707 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-credential-socket\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.362931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362767 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-envoy\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.362931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362791 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-socket\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.362931 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362875 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-data\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.363124 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362937 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-certs\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.363124 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362968 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-kube-api-access-frhgb\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-kube-api-access-frhgb\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.363124 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.362993 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-podinfo\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.363124 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.363088 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istiod-ca-cert\") pod \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\" (UID: \"6a1c20cf-b63c-4934-b0e9-485b2df551f4\") " Oct 31 22:18:11.363376 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.363209 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-socket" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:11.363575 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.363551 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istiod-ca-cert" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:11.363655 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.363631 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.363806 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.363786 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-credential-socket" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:11.363908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.363828 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-data" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:11.363908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.363846 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-certs" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:11.365798 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.365763 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-envoy" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:11.365904 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.365799 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-kube-api-access-frhgb" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "kube-api-access-frhgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:11.365904 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.365820 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-token" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:11.366026 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.365919 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-podinfo" pod "6a1c20cf-b63c-4934-b0e9-485b2df551f4" (UID: "6a1c20cf-b63c-4934-b0e9-485b2df551f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:18:11.464715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464688 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.464715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464713 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frhgb\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-kube-api-access-frhgb\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.464876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464728 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.464876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464742 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.464876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464753 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.464876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464765 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.464876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464776 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.464876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.464788 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a1c20cf-b63c-4934-b0e9-485b2df551f4-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.544921 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.544893 4294 generic.go:356] "Generic (PLEG): container finished" podID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerID="1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce" exitCode=0 Oct 31 22:18:11.545239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.544942 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" Oct 31 22:18:11.545239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.544975 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" event={"ID":"6a1c20cf-b63c-4934-b0e9-485b2df551f4","Type":"ContainerDied","Data":"1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce"} Oct 31 22:18:11.545239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.545035 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z" event={"ID":"6a1c20cf-b63c-4934-b0e9-485b2df551f4","Type":"ContainerDied","Data":"107f07a0fae215e5ddfc71965d6ed630820a621f4a5d6e5fe3aead48028ce6c3"} Oct 31 22:18:11.545239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.545055 4294 scope.go:122] "RemoveContainer" containerID="1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce" Oct 31 22:18:11.556369 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.556352 4294 scope.go:122] "RemoveContainer" containerID="f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347" Oct 31 22:18:11.566348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.566332 4294 scope.go:122] "RemoveContainer" containerID="1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce" Oct 31 22:18:11.566598 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:11.566577 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce\": container with ID starting with 1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce not found: ID does not exist" containerID="1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce" Oct 31 22:18:11.566713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.566600 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce"} err="failed to get container status \"1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce\": rpc error: code = NotFound desc = could not find container \"1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce\": container with ID starting with 1cfc105fb104a5db4bcff063a269a143b310c29f84802a4104949d2a493496ce not found: ID does not exist" Oct 31 22:18:11.566713 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.566619 4294 scope.go:122] "RemoveContainer" containerID="f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347" Oct 31 22:18:11.566875 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:11.566857 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347\": container with ID starting with f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347 not found: ID does not exist" containerID="f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347" Oct 31 22:18:11.566950 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.566876 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347"} err="failed to get container status \"f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347\": rpc error: code = NotFound desc = could not find container \"f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347\": container with ID starting with f6900c31f5b3ff95e28af09ce467f4ef64191d0454185388c40ca88777a0a347 not found: ID does not exist" Oct 31 22:18:11.597935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.597905 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z"] Oct 31 22:18:11.601850 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.601826 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c08360-openshift-default-848df6db77-dcc6z"] Oct 31 22:18:11.673218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.673196 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:11.768584 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.768558 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-openshift-ca\") pod \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " Oct 31 22:18:11.768772 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.768619 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-xks-ca\") pod \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " Oct 31 22:18:11.768772 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.768645 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1158c05b-ae26-4736-b2fd-a632bb7fdf65-kube-api-access-xbfml\" (UniqueName: \"kubernetes.io/projected/1158c05b-ae26-4736-b2fd-a632bb7fdf65-kube-api-access-xbfml\") pod \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\" (UID: \"1158c05b-ae26-4736-b2fd-a632bb7fdf65\") " Oct 31 22:18:11.768980 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.768957 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-openshift-ca" pod "1158c05b-ae26-4736-b2fd-a632bb7fdf65" (UID: "1158c05b-ae26-4736-b2fd-a632bb7fdf65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:11.770689 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.770670 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1158c05b-ae26-4736-b2fd-a632bb7fdf65-kube-api-access-xbfml" pod "1158c05b-ae26-4736-b2fd-a632bb7fdf65" (UID: "1158c05b-ae26-4736-b2fd-a632bb7fdf65"). InnerVolumeSpecName "kube-api-access-xbfml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:11.770747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.770685 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-xks-ca" pod "1158c05b-ae26-4736-b2fd-a632bb7fdf65" (UID: "1158c05b-ae26-4736-b2fd-a632bb7fdf65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:11.870286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.870262 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.870286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.870284 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1158c05b-ae26-4736-b2fd-a632bb7fdf65-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.870443 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.870293 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbfml\" (UniqueName: \"kubernetes.io/projected/1158c05b-ae26-4736-b2fd-a632bb7fdf65-kube-api-access-xbfml\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:11.984748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:11.984720 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" path="/var/lib/kubelet/pods/6a1c20cf-b63c-4934-b0e9-485b2df551f4/volumes" Oct 31 22:18:12.553928 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:12.553902 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" Oct 31 22:18:12.554405 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:12.553946 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2869d921-n9lhl" event={"ID":"1158c05b-ae26-4736-b2fd-a632bb7fdf65","Type":"ContainerDied","Data":"cd6e1d37f93268a551c1811568de47b612abdd7de0cd874067eba90b2a4bc515"} Oct 31 22:18:12.554405 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:12.554001 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd6e1d37f93268a551c1811568de47b612abdd7de0cd874067eba90b2a4bc515" Oct 31 22:18:17.415597 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:17.415565 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw"] Oct 31 22:18:17.428139 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:17.428109 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f1e791-ljxjw"] Oct 31 22:18:17.984111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:17.984082 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89bae720-5e95-4a31-8e6d-758190101956" path="/var/lib/kubelet/pods/89bae720-5e95-4a31-8e6d-758190101956/volumes" Oct 31 22:18:19.167739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.167701 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj"] Oct 31 22:18:19.541691 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.541641 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" podUID="b3972990-9501-4a33-97b9-84df70dcd649" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:19.541879 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.541802 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" podUID="04ac6d67-3ecb-4c43-9df3-b2a270278829" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:19.705067 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705032 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k"] Oct 31 22:18:19.705555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705539 4294 cpu_manager.go:414] "containerMap: removing container" podUID="900ff827-7636-42d2-9a5c-49c7a470e1a5" containerName="revoke-keys" Oct 31 22:18:19.705629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705556 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900ff827-7636-42d2-9a5c-49c7a470e1a5" containerName="revoke-keys" Oct 31 22:18:19.705629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705567 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" Oct 31 22:18:19.705629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705572 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" Oct 31 22:18:19.705629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705580 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" Oct 31 22:18:19.705629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705584 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" Oct 31 22:18:19.705629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705609 4294 cpu_manager.go:414] "containerMap: removing container" podUID="1158c05b-ae26-4736-b2fd-a632bb7fdf65" containerName="revoke-keys" Oct 31 22:18:19.705629 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705617 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1158c05b-ae26-4736-b2fd-a632bb7fdf65" containerName="revoke-keys" Oct 31 22:18:19.705985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705742 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="900ff827-7636-42d2-9a5c-49c7a470e1a5" containerName="revoke-keys" Oct 31 22:18:19.705985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705753 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" Oct 31 22:18:19.705985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705764 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1c20cf-b63c-4934-b0e9-485b2df551f4" containerName="istio-proxy" Oct 31 22:18:19.705985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.705775 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="1158c05b-ae26-4736-b2fd-a632bb7fdf65" containerName="revoke-keys" Oct 31 22:18:19.710287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.710266 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.728733 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.728679 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k"] Oct 31 22:18:19.762880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.762852 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.762988 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.762930 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.762988 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.762957 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjrn\" (UniqueName: \"kubernetes.io/projected/c8d09e7d-a810-4971-a875-048398ed219c-kube-api-access-xmjrn\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.822030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.821954 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:18:19.822030 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.821977 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:18:19.863885 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.863857 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.863995 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.863924 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.863995 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.863946 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjrn\" (UniqueName: \"kubernetes.io/projected/c8d09e7d-a810-4971-a875-048398ed219c-kube-api-access-xmjrn\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.864610 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.864587 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.866106 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.866089 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:19.888117 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:19.888087 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjrn\" (UniqueName: \"kubernetes.io/projected/c8d09e7d-a810-4971-a875-048398ed219c-kube-api-access-xmjrn\") pod \"maas-api-revoke-sub-38ac7ac6-cn68k\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:20.023236 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:20.023210 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:20.323996 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:20.323963 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k"] Oct 31 22:18:20.325311 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:18:20.325219 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d09e7d_a810_4971_a875_048398ed219c.slice/crio-e2823d27702de4e517a0fe8ba833bdcafa9397e3c5e9f6d9e2afed86d7309727 WatchSource:0}: Error finding container e2823d27702de4e517a0fe8ba833bdcafa9397e3c5e9f6d9e2afed86d7309727: Status 404 returned error can't find the container with id e2823d27702de4e517a0fe8ba833bdcafa9397e3c5e9f6d9e2afed86d7309727 Oct 31 22:18:20.611186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:20.611160 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" event={"ID":"c8d09e7d-a810-4971-a875-048398ed219c","Type":"ContainerStarted","Data":"2b5136322b29648dd1cc071fdf74684390cd7c05040bb29ed62d99d88e2fd10d"} Oct 31 22:18:20.611344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:20.611211 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" event={"ID":"c8d09e7d-a810-4971-a875-048398ed219c","Type":"ContainerStarted","Data":"e2823d27702de4e517a0fe8ba833bdcafa9397e3c5e9f6d9e2afed86d7309727"} Oct 31 22:18:20.655244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:20.655195 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" podStartSLOduration=1.655182631 podStartE2EDuration="1.655182631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:18:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:18:20.65344873 +0000 UTC m=+3740.778555961" watchObservedRunningTime="2026-10-31 22:18:20.655182631 +0000 UTC m=+3740.780289862" Oct 31 22:18:21.618571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:21.618542 4294 generic.go:356] "Generic (PLEG): container finished" podID="c8d09e7d-a810-4971-a875-048398ed219c" containerID="2b5136322b29648dd1cc071fdf74684390cd7c05040bb29ed62d99d88e2fd10d" exitCode=0 Oct 31 22:18:21.618956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:21.618628 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" event={"ID":"c8d09e7d-a810-4971-a875-048398ed219c","Type":"ContainerDied","Data":"2b5136322b29648dd1cc071fdf74684390cd7c05040bb29ed62d99d88e2fd10d"} Oct 31 22:18:22.473817 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.473791 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 22:18:22.489600 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.489574 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 22:18:22.590889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.590852 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3972990-9501-4a33-97b9-84df70dcd649-kube-api-access-k6rtd\" (UniqueName: \"kubernetes.io/projected/b3972990-9501-4a33-97b9-84df70dcd649-kube-api-access-k6rtd\") pod \"b3972990-9501-4a33-97b9-84df70dcd649\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " Oct 31 22:18:22.591057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.590995 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04ac6d67-3ecb-4c43-9df3-b2a270278829-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ac6d67-3ecb-4c43-9df3-b2a270278829-plugins-config-volume\") pod \"04ac6d67-3ecb-4c43-9df3-b2a270278829\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " Oct 31 22:18:22.591057 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.591033 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3972990-9501-4a33-97b9-84df70dcd649-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3972990-9501-4a33-97b9-84df70dcd649-plugins-config-volume\") pod \"b3972990-9501-4a33-97b9-84df70dcd649\" (UID: \"b3972990-9501-4a33-97b9-84df70dcd649\") " Oct 31 22:18:22.591224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.591079 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04ac6d67-3ecb-4c43-9df3-b2a270278829-kube-api-access-r7mmn\" (UniqueName: \"kubernetes.io/projected/04ac6d67-3ecb-4c43-9df3-b2a270278829-kube-api-access-r7mmn\") pod \"04ac6d67-3ecb-4c43-9df3-b2a270278829\" (UID: \"04ac6d67-3ecb-4c43-9df3-b2a270278829\") " Oct 31 22:18:22.591462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.591436 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3972990-9501-4a33-97b9-84df70dcd649-plugins-config-volume" pod "b3972990-9501-4a33-97b9-84df70dcd649" (UID: "b3972990-9501-4a33-97b9-84df70dcd649"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:22.591562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.591457 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ac6d67-3ecb-4c43-9df3-b2a270278829-plugins-config-volume" pod "04ac6d67-3ecb-4c43-9df3-b2a270278829" (UID: "04ac6d67-3ecb-4c43-9df3-b2a270278829"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:22.591779 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.591750 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04ac6d67-3ecb-4c43-9df3-b2a270278829-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:22.591968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.591783 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3972990-9501-4a33-97b9-84df70dcd649-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:22.593348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.593294 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3972990-9501-4a33-97b9-84df70dcd649-kube-api-access-k6rtd" pod "b3972990-9501-4a33-97b9-84df70dcd649" (UID: "b3972990-9501-4a33-97b9-84df70dcd649"). InnerVolumeSpecName "kube-api-access-k6rtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:22.593348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.593332 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ac6d67-3ecb-4c43-9df3-b2a270278829-kube-api-access-r7mmn" pod "04ac6d67-3ecb-4c43-9df3-b2a270278829" (UID: "04ac6d67-3ecb-4c43-9df3-b2a270278829"). InnerVolumeSpecName "kube-api-access-r7mmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:22.626185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.626158 4294 generic.go:356] "Generic (PLEG): container finished" podID="b3972990-9501-4a33-97b9-84df70dcd649" containerID="3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca" exitCode=0 Oct 31 22:18:22.626556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.626206 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" Oct 31 22:18:22.626556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.626236 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" event={"ID":"b3972990-9501-4a33-97b9-84df70dcd649","Type":"ContainerDied","Data":"3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca"} Oct 31 22:18:22.626556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.626285 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf" event={"ID":"b3972990-9501-4a33-97b9-84df70dcd649","Type":"ContainerDied","Data":"9cc26e762785c1b5fa2b84fa0f4196389dee78520de19f17087394b8b8cdadcb"} Oct 31 22:18:22.626556 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.626318 4294 scope.go:122] "RemoveContainer" containerID="3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca" Oct 31 22:18:22.627669 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.627650 4294 generic.go:356] "Generic (PLEG): container finished" podID="04ac6d67-3ecb-4c43-9df3-b2a270278829" containerID="beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c" exitCode=0 Oct 31 22:18:22.627747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.627710 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" Oct 31 22:18:22.627747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.627724 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" event={"ID":"04ac6d67-3ecb-4c43-9df3-b2a270278829","Type":"ContainerDied","Data":"beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c"} Oct 31 22:18:22.627842 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.627777 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw" event={"ID":"04ac6d67-3ecb-4c43-9df3-b2a270278829","Type":"ContainerDied","Data":"3cfd45a23c569ec9b144a0308340380f389ba58e1d1cacd2f72e8455b7074963"} Oct 31 22:18:22.644729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.644711 4294 scope.go:122] "RemoveContainer" containerID="3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca" Oct 31 22:18:22.645040 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:22.645011 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca\": container with ID starting with 3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca not found: ID does not exist" containerID="3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca" Oct 31 22:18:22.645106 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.645039 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca"} err="failed to get container status \"3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca\": rpc error: code = NotFound desc = could not find container \"3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca\": container with ID starting with 3c6efa956aae76a59d2f10d4db6aba4ea9ef6e3bc80a99f04e1ecc38375d38ca not found: ID does not exist" Oct 31 22:18:22.645106 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.645056 4294 scope.go:122] "RemoveContainer" containerID="beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c" Oct 31 22:18:22.655946 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.655923 4294 scope.go:122] "RemoveContainer" containerID="beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c" Oct 31 22:18:22.656271 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:22.656246 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c\": container with ID starting with beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c not found: ID does not exist" containerID="beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c" Oct 31 22:18:22.656385 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.656277 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c"} err="failed to get container status \"beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c\": rpc error: code = NotFound desc = could not find container \"beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c\": container with ID starting with beeacbbad9f33fa18247becf6ecf37bb9030352b7b51a60548c4681b58787d5c not found: ID does not exist" Oct 31 22:18:22.682627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.682597 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf"] Oct 31 22:18:22.708513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.706591 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f1e791-5f7f56dfff-j78zf"] Oct 31 22:18:22.708513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.707575 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rtd\" (UniqueName: \"kubernetes.io/projected/b3972990-9501-4a33-97b9-84df70dcd649-kube-api-access-k6rtd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:22.708513 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.707767 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mmn\" (UniqueName: \"kubernetes.io/projected/04ac6d67-3ecb-4c43-9df3-b2a270278829-kube-api-access-r7mmn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:22.730956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.730920 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw"] Oct 31 22:18:22.736739 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.736704 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f1e791-7fdc98cb9f-c9dzw"] Oct 31 22:18:22.768873 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.768851 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:22.909914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.909886 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-openshift-ca\") pod \"c8d09e7d-a810-4971-a875-048398ed219c\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " Oct 31 22:18:22.910082 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.909977 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-xks-ca\") pod \"c8d09e7d-a810-4971-a875-048398ed219c\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " Oct 31 22:18:22.910082 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.910016 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d09e7d-a810-4971-a875-048398ed219c-kube-api-access-xmjrn\" (UniqueName: \"kubernetes.io/projected/c8d09e7d-a810-4971-a875-048398ed219c-kube-api-access-xmjrn\") pod \"c8d09e7d-a810-4971-a875-048398ed219c\" (UID: \"c8d09e7d-a810-4971-a875-048398ed219c\") " Oct 31 22:18:22.910223 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.910202 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-openshift-ca" pod "c8d09e7d-a810-4971-a875-048398ed219c" (UID: "c8d09e7d-a810-4971-a875-048398ed219c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:22.910313 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.910289 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:22.911867 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.911846 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-xks-ca" pod "c8d09e7d-a810-4971-a875-048398ed219c" (UID: "c8d09e7d-a810-4971-a875-048398ed219c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:22.912002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:22.911976 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d09e7d-a810-4971-a875-048398ed219c-kube-api-access-xmjrn" pod "c8d09e7d-a810-4971-a875-048398ed219c" (UID: "c8d09e7d-a810-4971-a875-048398ed219c"). InnerVolumeSpecName "kube-api-access-xmjrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:23.011580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:23.011561 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8d09e7d-a810-4971-a875-048398ed219c-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:23.011580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:23.011580 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjrn\" (UniqueName: \"kubernetes.io/projected/c8d09e7d-a810-4971-a875-048398ed219c-kube-api-access-xmjrn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:23.644709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:23.644661 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" event={"ID":"c8d09e7d-a810-4971-a875-048398ed219c","Type":"ContainerDied","Data":"e2823d27702de4e517a0fe8ba833bdcafa9397e3c5e9f6d9e2afed86d7309727"} Oct 31 22:18:23.645157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:23.644772 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2823d27702de4e517a0fe8ba833bdcafa9397e3c5e9f6d9e2afed86d7309727" Oct 31 22:18:23.645157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:23.644674 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ac7ac6-cn68k" Oct 31 22:18:23.984438 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:23.984402 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04ac6d67-3ecb-4c43-9df3-b2a270278829" path="/var/lib/kubelet/pods/04ac6d67-3ecb-4c43-9df3-b2a270278829/volumes" Oct 31 22:18:23.984783 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:23.984769 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3972990-9501-4a33-97b9-84df70dcd649" path="/var/lib/kubelet/pods/b3972990-9501-4a33-97b9-84df70dcd649/volumes" Oct 31 22:18:25.103381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103348 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l"] Oct 31 22:18:25.103785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103772 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c8d09e7d-a810-4971-a875-048398ed219c" containerName="revoke-keys" Oct 31 22:18:25.103785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103785 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d09e7d-a810-4971-a875-048398ed219c" containerName="revoke-keys" Oct 31 22:18:25.103900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103798 4294 cpu_manager.go:414] "containerMap: removing container" podUID="04ac6d67-3ecb-4c43-9df3-b2a270278829" containerName="payload-pre-processing" Oct 31 22:18:25.103900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103804 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04ac6d67-3ecb-4c43-9df3-b2a270278829" containerName="payload-pre-processing" Oct 31 22:18:25.103900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103813 4294 cpu_manager.go:414] "containerMap: removing container" podUID="b3972990-9501-4a33-97b9-84df70dcd649" containerName="payload-processing" Oct 31 22:18:25.103900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103817 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3972990-9501-4a33-97b9-84df70dcd649" containerName="payload-processing" Oct 31 22:18:25.104044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103910 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d09e7d-a810-4971-a875-048398ed219c" containerName="revoke-keys" Oct 31 22:18:25.104044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103918 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="04ac6d67-3ecb-4c43-9df3-b2a270278829" containerName="payload-pre-processing" Oct 31 22:18:25.104044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.103923 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3972990-9501-4a33-97b9-84df70dcd649" containerName="payload-processing" Oct 31 22:18:25.108478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.108460 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.129006 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.128980 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l"] Oct 31 22:18:25.176830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.176800 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:18:25.176830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.176811 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:18:25.238175 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.238136 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.238361 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.238332 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.238422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.238371 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgwp\" (UniqueName: \"kubernetes.io/projected/6e2f9f3f-6a02-4f65-be06-f62bcf092644-kube-api-access-rqgwp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.340119 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.340080 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.340285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.340243 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.340346 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.340283 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgwp\" (UniqueName: \"kubernetes.io/projected/6e2f9f3f-6a02-4f65-be06-f62bcf092644-kube-api-access-rqgwp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.340846 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.340822 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.342992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.342944 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.365238 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.365172 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgwp\" (UniqueName: \"kubernetes.io/projected/6e2f9f3f-6a02-4f65-be06-f62bcf092644-kube-api-access-rqgwp\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.421697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.421666 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:25.727148 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:25.727123 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l"] Oct 31 22:18:25.729579 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:18:25.729546 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2f9f3f_6a02_4f65_be06_f62bcf092644.slice/crio-f1ded4ff0b3aec97b48bc0d77b5070c2a958198a6920d4683bb00791166f6de4 WatchSource:0}: Error finding container f1ded4ff0b3aec97b48bc0d77b5070c2a958198a6920d4683bb00791166f6de4: Status 404 returned error can't find the container with id f1ded4ff0b3aec97b48bc0d77b5070c2a958198a6920d4683bb00791166f6de4 Oct 31 22:18:26.704551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:26.704522 4294 generic.go:356] "Generic (PLEG): container finished" podID="6e2f9f3f-6a02-4f65-be06-f62bcf092644" containerID="016a4022da6b458bac1fec5df8b6a433fe8ed2dc242bee188158f9d668e4fb7b" exitCode=0 Oct 31 22:18:26.704907 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:26.704612 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" event={"ID":"6e2f9f3f-6a02-4f65-be06-f62bcf092644","Type":"ContainerDied","Data":"016a4022da6b458bac1fec5df8b6a433fe8ed2dc242bee188158f9d668e4fb7b"} Oct 31 22:18:26.704907 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:26.704685 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" event={"ID":"6e2f9f3f-6a02-4f65-be06-f62bcf092644","Type":"ContainerStarted","Data":"f1ded4ff0b3aec97b48bc0d77b5070c2a958198a6920d4683bb00791166f6de4"} Oct 31 22:18:27.853689 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.853660 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:27.969499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.969411 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-xks-ca\") pod \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " Oct 31 22:18:27.969499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.969495 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2f9f3f-6a02-4f65-be06-f62bcf092644-kube-api-access-rqgwp\" (UniqueName: \"kubernetes.io/projected/6e2f9f3f-6a02-4f65-be06-f62bcf092644-kube-api-access-rqgwp\") pod \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " Oct 31 22:18:27.969748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.969518 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-openshift-ca\") pod \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\" (UID: \"6e2f9f3f-6a02-4f65-be06-f62bcf092644\") " Oct 31 22:18:27.969968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.969937 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-openshift-ca" pod "6e2f9f3f-6a02-4f65-be06-f62bcf092644" (UID: "6e2f9f3f-6a02-4f65-be06-f62bcf092644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:27.970135 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.970113 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:27.971861 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.971834 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2f9f3f-6a02-4f65-be06-f62bcf092644-kube-api-access-rqgwp" pod "6e2f9f3f-6a02-4f65-be06-f62bcf092644" (UID: "6e2f9f3f-6a02-4f65-be06-f62bcf092644"). InnerVolumeSpecName "kube-api-access-rqgwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:27.973037 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:27.973014 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-xks-ca" pod "6e2f9f3f-6a02-4f65-be06-f62bcf092644" (UID: "6e2f9f3f-6a02-4f65-be06-f62bcf092644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:28.071773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:28.071732 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqgwp\" (UniqueName: \"kubernetes.io/projected/6e2f9f3f-6a02-4f65-be06-f62bcf092644-kube-api-access-rqgwp\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:28.071773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:28.071773 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e2f9f3f-6a02-4f65-be06-f62bcf092644-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:28.721729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:28.721645 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" Oct 31 22:18:28.721729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:28.721659 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l" event={"ID":"6e2f9f3f-6a02-4f65-be06-f62bcf092644","Type":"ContainerDied","Data":"f1ded4ff0b3aec97b48bc0d77b5070c2a958198a6920d4683bb00791166f6de4"} Oct 31 22:18:28.722056 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:28.722007 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1ded4ff0b3aec97b48bc0d77b5070c2a958198a6920d4683bb00791166f6de4" Oct 31 22:18:30.031797 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:30.031760 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k"] Oct 31 22:18:30.044185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:30.044156 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3d8942b-hh79k"] Oct 31 22:18:31.309731 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:31.309694 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:18:31.891935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:31.891804 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs"] Oct 31 22:18:32.016563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.016523 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4eef1f7-82e5-4abf-b44e-8733b2d3a294" path="/var/lib/kubelet/pods/e4eef1f7-82e5-4abf-b44e-8733b2d3a294/volumes" Oct 31 22:18:32.192769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.192692 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7"] Oct 31 22:18:32.203342 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.203288 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-696f9a-q72p7"] Oct 31 22:18:32.325467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.325437 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5"] Oct 31 22:18:32.325900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.325888 4294 cpu_manager.go:414] "containerMap: removing container" podUID="6e2f9f3f-6a02-4f65-be06-f62bcf092644" containerName="revoke-keys" Oct 31 22:18:32.325940 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.325901 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2f9f3f-6a02-4f65-be06-f62bcf092644" containerName="revoke-keys" Oct 31 22:18:32.326014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.326005 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2f9f3f-6a02-4f65-be06-f62bcf092644" containerName="revoke-keys" Oct 31 22:18:32.330429 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.330405 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.342314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.342281 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f747w\" (UniqueName: \"kubernetes.io/projected/edbca627-1eb9-4242-baee-9817a22473e2-kube-api-access-f747w\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.342456 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.342434 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.342564 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.342548 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.346613 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.346590 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:18:32.346728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.346658 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:18:32.349091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.349070 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5"] Oct 31 22:18:32.443680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.443618 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f747w\" (UniqueName: \"kubernetes.io/projected/edbca627-1eb9-4242-baee-9817a22473e2-kube-api-access-f747w\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.443680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.443677 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.443882 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.443740 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.444287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.444268 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.446202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.446180 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.468088 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.468067 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f747w\" (UniqueName: \"kubernetes.io/projected/edbca627-1eb9-4242-baee-9817a22473e2-kube-api-access-f747w\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.642876 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.642852 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:32.940313 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:32.940283 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5"] Oct 31 22:18:32.942260 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:18:32.942237 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbca627_1eb9_4242_baee_9817a22473e2.slice/crio-f166acb63ecf278f49353aa61aeadd04b1baa64035401446c47bc9ef4f9a54ea WatchSource:0}: Error finding container f166acb63ecf278f49353aa61aeadd04b1baa64035401446c47bc9ef4f9a54ea: Status 404 returned error can't find the container with id f166acb63ecf278f49353aa61aeadd04b1baa64035401446c47bc9ef4f9a54ea Oct 31 22:18:33.756092 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:33.756060 4294 generic.go:356] "Generic (PLEG): container finished" podID="edbca627-1eb9-4242-baee-9817a22473e2" containerID="b44add3cc04431dcd7ef5b9ac8d07ea4321f5cc01f15a42f3004cf2323d6cad6" exitCode=0 Oct 31 22:18:33.756553 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:33.756138 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" event={"ID":"edbca627-1eb9-4242-baee-9817a22473e2","Type":"ContainerDied","Data":"b44add3cc04431dcd7ef5b9ac8d07ea4321f5cc01f15a42f3004cf2323d6cad6"} Oct 31 22:18:33.756553 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:33.756184 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" event={"ID":"edbca627-1eb9-4242-baee-9817a22473e2","Type":"ContainerStarted","Data":"f166acb63ecf278f49353aa61aeadd04b1baa64035401446c47bc9ef4f9a54ea"} Oct 31 22:18:33.984630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:33.984589 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900ff827-7636-42d2-9a5c-49c7a470e1a5" path="/var/lib/kubelet/pods/900ff827-7636-42d2-9a5c-49c7a470e1a5/volumes" Oct 31 22:18:34.191598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.191562 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" podUID="bfde9ff9-1991-43e3-8e62-99c461f90862" containerName="main" containerID="cri-o://db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f" gracePeriod=45 Oct 31 22:18:34.498217 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.498186 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:18:34.672649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672575 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-tmp-dir\") pod \"bfde9ff9-1991-43e3-8e62-99c461f90862\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " Oct 31 22:18:34.672649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672628 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-model-cache\") pod \"bfde9ff9-1991-43e3-8e62-99c461f90862\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " Oct 31 22:18:34.672839 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672719 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfde9ff9-1991-43e3-8e62-99c461f90862-kube-api-access-s88gx\" (UniqueName: \"kubernetes.io/projected/bfde9ff9-1991-43e3-8e62-99c461f90862-kube-api-access-s88gx\") pod \"bfde9ff9-1991-43e3-8e62-99c461f90862\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " Oct 31 22:18:34.672839 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672751 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-home\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-home\") pod \"bfde9ff9-1991-43e3-8e62-99c461f90862\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " Oct 31 22:18:34.672839 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672790 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-dshm\") pod \"bfde9ff9-1991-43e3-8e62-99c461f90862\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " Oct 31 22:18:34.672839 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672833 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfde9ff9-1991-43e3-8e62-99c461f90862-tls-certs\" (UniqueName: \"kubernetes.io/secret/bfde9ff9-1991-43e3-8e62-99c461f90862-tls-certs\") pod \"bfde9ff9-1991-43e3-8e62-99c461f90862\" (UID: \"bfde9ff9-1991-43e3-8e62-99c461f90862\") " Oct 31 22:18:34.672997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672854 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-tmp-dir" pod "bfde9ff9-1991-43e3-8e62-99c461f90862" (UID: "bfde9ff9-1991-43e3-8e62-99c461f90862"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:34.672997 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.672890 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-model-cache" pod "bfde9ff9-1991-43e3-8e62-99c461f90862" (UID: "bfde9ff9-1991-43e3-8e62-99c461f90862"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:34.673087 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.673023 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-home" pod "bfde9ff9-1991-43e3-8e62-99c461f90862" (UID: "bfde9ff9-1991-43e3-8e62-99c461f90862"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:34.673295 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.673277 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:34.673418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.673297 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:34.673418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.673339 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:34.675051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.675026 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-dshm" pod "bfde9ff9-1991-43e3-8e62-99c461f90862" (UID: "bfde9ff9-1991-43e3-8e62-99c461f90862"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:34.675154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.675052 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfde9ff9-1991-43e3-8e62-99c461f90862-tls-certs" pod "bfde9ff9-1991-43e3-8e62-99c461f90862" (UID: "bfde9ff9-1991-43e3-8e62-99c461f90862"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:34.675154 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.675074 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfde9ff9-1991-43e3-8e62-99c461f90862-kube-api-access-s88gx" pod "bfde9ff9-1991-43e3-8e62-99c461f90862" (UID: "bfde9ff9-1991-43e3-8e62-99c461f90862"). InnerVolumeSpecName "kube-api-access-s88gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:34.764169 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.764142 4294 generic.go:356] "Generic (PLEG): container finished" podID="bfde9ff9-1991-43e3-8e62-99c461f90862" containerID="db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f" exitCode=0 Oct 31 22:18:34.764529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.764187 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" Oct 31 22:18:34.764529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.764229 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" event={"ID":"bfde9ff9-1991-43e3-8e62-99c461f90862","Type":"ContainerDied","Data":"db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f"} Oct 31 22:18:34.764529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.764272 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj" event={"ID":"bfde9ff9-1991-43e3-8e62-99c461f90862","Type":"ContainerDied","Data":"f84aa22481d935ba5187dcf4c2d7761d6fcdf51d4201ea9e7a0565b91f930527"} Oct 31 22:18:34.764529 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.764289 4294 scope.go:122] "RemoveContainer" containerID="db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f" Oct 31 22:18:34.775283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.775156 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s88gx\" (UniqueName: \"kubernetes.io/projected/bfde9ff9-1991-43e3-8e62-99c461f90862-kube-api-access-s88gx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:34.775283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.775181 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfde9ff9-1991-43e3-8e62-99c461f90862-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:34.775283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.775196 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfde9ff9-1991-43e3-8e62-99c461f90862-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:34.776262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.776247 4294 scope.go:122] "RemoveContainer" containerID="db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f" Oct 31 22:18:34.776653 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:34.776634 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f\": container with ID starting with db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f not found: ID does not exist" containerID="db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f" Oct 31 22:18:34.776718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.776661 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f"} err="failed to get container status \"db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f\": rpc error: code = NotFound desc = could not find container \"db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f\": container with ID starting with db49d2b8438091f51759d527f2677447d98d7e4f5d4104cd75e7f9bab234302f not found: ID does not exist" Oct 31 22:18:34.819780 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.819742 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj"] Oct 31 22:18:34.832377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.832353 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1b0cce/test-model-1b0cce-kserve-76d5d847f9-p52lj"] Oct 31 22:18:34.910292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:34.910271 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:35.078780 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.078755 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-openshift-ca\") pod \"edbca627-1eb9-4242-baee-9817a22473e2\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " Oct 31 22:18:35.078953 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.078848 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-xks-ca\") pod \"edbca627-1eb9-4242-baee-9817a22473e2\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " Oct 31 22:18:35.078953 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.078929 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edbca627-1eb9-4242-baee-9817a22473e2-kube-api-access-f747w\" (UniqueName: \"kubernetes.io/projected/edbca627-1eb9-4242-baee-9817a22473e2-kube-api-access-f747w\") pod \"edbca627-1eb9-4242-baee-9817a22473e2\" (UID: \"edbca627-1eb9-4242-baee-9817a22473e2\") " Oct 31 22:18:35.079176 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.079153 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-openshift-ca" pod "edbca627-1eb9-4242-baee-9817a22473e2" (UID: "edbca627-1eb9-4242-baee-9817a22473e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:35.080859 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.080834 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbca627-1eb9-4242-baee-9817a22473e2-kube-api-access-f747w" pod "edbca627-1eb9-4242-baee-9817a22473e2" (UID: "edbca627-1eb9-4242-baee-9817a22473e2"). InnerVolumeSpecName "kube-api-access-f747w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:35.080859 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.080846 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-xks-ca" pod "edbca627-1eb9-4242-baee-9817a22473e2" (UID: "edbca627-1eb9-4242-baee-9817a22473e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:35.180072 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.180046 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:35.180072 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.180070 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/edbca627-1eb9-4242-baee-9817a22473e2-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:35.180242 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.180080 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f747w\" (UniqueName: \"kubernetes.io/projected/edbca627-1eb9-4242-baee-9817a22473e2-kube-api-access-f747w\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:35.561729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.561693 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6"] Oct 31 22:18:35.597627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.597594 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6"] Oct 31 22:18:35.772497 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.772465 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" event={"ID":"edbca627-1eb9-4242-baee-9817a22473e2","Type":"ContainerDied","Data":"f166acb63ecf278f49353aa61aeadd04b1baa64035401446c47bc9ef4f9a54ea"} Oct 31 22:18:35.772971 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.772497 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5" Oct 31 22:18:35.772971 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.772512 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f166acb63ecf278f49353aa61aeadd04b1baa64035401446c47bc9ef4f9a54ea" Oct 31 22:18:35.986422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:35.986391 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfde9ff9-1991-43e3-8e62-99c461f90862" path="/var/lib/kubelet/pods/bfde9ff9-1991-43e3-8e62-99c461f90862/volumes" Oct 31 22:18:39.265838 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.265801 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb"] Oct 31 22:18:39.266477 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.266043 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" containerID="cri-o://eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52" gracePeriod=30 Oct 31 22:18:39.866743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.866710 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889"] Oct 31 22:18:39.867208 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.867194 4294 cpu_manager.go:414] "containerMap: removing container" podUID="edbca627-1eb9-4242-baee-9817a22473e2" containerName="revoke-keys" Oct 31 22:18:39.867268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.867209 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edbca627-1eb9-4242-baee-9817a22473e2" containerName="revoke-keys" Oct 31 22:18:39.867268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.867225 4294 cpu_manager.go:414] "containerMap: removing container" podUID="bfde9ff9-1991-43e3-8e62-99c461f90862" containerName="main" Oct 31 22:18:39.867268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.867230 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfde9ff9-1991-43e3-8e62-99c461f90862" containerName="main" Oct 31 22:18:39.867422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.867323 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="edbca627-1eb9-4242-baee-9817a22473e2" containerName="revoke-keys" Oct 31 22:18:39.867422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.867336 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfde9ff9-1991-43e3-8e62-99c461f90862" containerName="main" Oct 31 22:18:39.870641 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.870623 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:39.889765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.889743 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889"] Oct 31 22:18:39.899391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.899369 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:18:39.899510 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.899436 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:18:39.929709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.929683 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:39.929809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.929732 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nswc\" (UniqueName: \"kubernetes.io/projected/0afe9610-ed31-4a99-8a83-f87ab0abb694-kube-api-access-7nswc\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:39.929871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:39.929851 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.030547 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.030521 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.030666 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.030574 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nswc\" (UniqueName: \"kubernetes.io/projected/0afe9610-ed31-4a99-8a83-f87ab0abb694-kube-api-access-7nswc\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.030666 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.030623 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.031219 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.031202 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.032826 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.032806 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.054442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.054423 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nswc\" (UniqueName: \"kubernetes.io/projected/0afe9610-ed31-4a99-8a83-f87ab0abb694-kube-api-access-7nswc\") pod \"maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.184314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.184232 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:40.483114 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.483080 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889"] Oct 31 22:18:40.484802 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:18:40.484764 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afe9610_ed31_4a99_8a83_f87ab0abb694.slice/crio-d7ee87426948e925dee452e7cecbc857d2bd166fdb9c53692763508ca3239df4 WatchSource:0}: Error finding container d7ee87426948e925dee452e7cecbc857d2bd166fdb9c53692763508ca3239df4: Status 404 returned error can't find the container with id d7ee87426948e925dee452e7cecbc857d2bd166fdb9c53692763508ca3239df4 Oct 31 22:18:40.562183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.562153 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" podUID="835a2433-6d43-4795-b33f-3a8f3bbf586d" containerName="payload-processing" containerID="cri-o://2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258" gracePeriod=25 Oct 31 22:18:40.598712 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.598684 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" podUID="97f1ea75-483a-44a8-aab9-ce156938abe5" containerName="payload-pre-processing" containerID="cri-o://2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd" gracePeriod=25 Oct 31 22:18:40.804707 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.804674 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" event={"ID":"0afe9610-ed31-4a99-8a83-f87ab0abb694","Type":"ContainerStarted","Data":"35a6cdca6acc40015d92d1b1725beb821ab9adf16c9b293ea672f028f04ba2d4"} Oct 31 22:18:40.804819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.804732 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" event={"ID":"0afe9610-ed31-4a99-8a83-f87ab0abb694","Type":"ContainerStarted","Data":"d7ee87426948e925dee452e7cecbc857d2bd166fdb9c53692763508ca3239df4"} Oct 31 22:18:40.849116 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:40.849074 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" podStartSLOduration=1.84905958 podStartE2EDuration="1.84905958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:18:40.846738128 +0000 UTC m=+3760.971845358" watchObservedRunningTime="2026-10-31 22:18:40.84905958 +0000 UTC m=+3760.974166844" Oct 31 22:18:41.031258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:41.031180 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss"] Oct 31 22:18:41.034080 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:41.034058 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b7fab9f6-7jtss"] Oct 31 22:18:41.812621 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:41.812586 4294 generic.go:356] "Generic (PLEG): container finished" podID="0afe9610-ed31-4a99-8a83-f87ab0abb694" containerID="35a6cdca6acc40015d92d1b1725beb821ab9adf16c9b293ea672f028f04ba2d4" exitCode=0 Oct 31 22:18:41.813023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:41.812647 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" event={"ID":"0afe9610-ed31-4a99-8a83-f87ab0abb694","Type":"ContainerDied","Data":"35a6cdca6acc40015d92d1b1725beb821ab9adf16c9b293ea672f028f04ba2d4"} Oct 31 22:18:41.983886 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:41.983850 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed36186-d38a-4289-99ea-9fa780758130" path="/var/lib/kubelet/pods/eed36186-d38a-4289-99ea-9fa780758130/volumes" Oct 31 22:18:42.994541 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:42.994517 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:43.070990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.070902 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-openshift-ca\") pod \"0afe9610-ed31-4a99-8a83-f87ab0abb694\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " Oct 31 22:18:43.070990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.070971 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afe9610-ed31-4a99-8a83-f87ab0abb694-kube-api-access-7nswc\" (UniqueName: \"kubernetes.io/projected/0afe9610-ed31-4a99-8a83-f87ab0abb694-kube-api-access-7nswc\") pod \"0afe9610-ed31-4a99-8a83-f87ab0abb694\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " Oct 31 22:18:43.071251 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.071072 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-xks-ca\") pod \"0afe9610-ed31-4a99-8a83-f87ab0abb694\" (UID: \"0afe9610-ed31-4a99-8a83-f87ab0abb694\") " Oct 31 22:18:43.071408 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.071374 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-openshift-ca" pod "0afe9610-ed31-4a99-8a83-f87ab0abb694" (UID: "0afe9610-ed31-4a99-8a83-f87ab0abb694"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:43.071545 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.071493 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:43.073150 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.073125 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afe9610-ed31-4a99-8a83-f87ab0abb694-kube-api-access-7nswc" pod "0afe9610-ed31-4a99-8a83-f87ab0abb694" (UID: "0afe9610-ed31-4a99-8a83-f87ab0abb694"). InnerVolumeSpecName "kube-api-access-7nswc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:43.073269 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.073175 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-xks-ca" pod "0afe9610-ed31-4a99-8a83-f87ab0abb694" (UID: "0afe9610-ed31-4a99-8a83-f87ab0abb694"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:43.172427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.172397 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afe9610-ed31-4a99-8a83-f87ab0abb694-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:43.172427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.172429 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nswc\" (UniqueName: \"kubernetes.io/projected/0afe9610-ed31-4a99-8a83-f87ab0abb694-kube-api-access-7nswc\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:43.826516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.826487 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" Oct 31 22:18:43.826516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.826502 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889" event={"ID":"0afe9610-ed31-4a99-8a83-f87ab0abb694","Type":"ContainerDied","Data":"d7ee87426948e925dee452e7cecbc857d2bd166fdb9c53692763508ca3239df4"} Oct 31 22:18:43.826729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:43.826552 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ee87426948e925dee452e7cecbc857d2bd166fdb9c53692763508ca3239df4" Oct 31 22:18:44.537469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.537415 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 22:18:44.588573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588542 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-envoy\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.588721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588608 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-podinfo\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.588721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588640 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-certs\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.588721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588685 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-token\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-token\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.588875 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588730 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45584a2a-6fd4-45fe-8b86-6add93774c95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45584a2a-6fd4-45fe-8b86-6add93774c95-istiod-ca-cert\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.588875 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588773 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-kube-api-access-dcd7x\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-kube-api-access-dcd7x\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.588984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588886 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-socket\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.588984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588935 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-data\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.589111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588952 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-certs" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:44.589111 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.588967 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-credential-socket\") pod \"45584a2a-6fd4-45fe-8b86-6add93774c95\" (UID: \"45584a2a-6fd4-45fe-8b86-6add93774c95\") " Oct 31 22:18:44.589258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589236 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-credential-socket" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:44.589258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589251 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-socket" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:44.589442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589415 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-data" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:44.589442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589428 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45584a2a-6fd4-45fe-8b86-6add93774c95-istiod-ca-cert" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:18:44.589804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589777 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.589804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589801 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.589977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589815 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.589977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589829 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.589977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.589843 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45584a2a-6fd4-45fe-8b86-6add93774c95-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.591075 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.591051 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-token" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:44.591183 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.591139 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-podinfo" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:18:44.591409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.591386 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-envoy" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:44.591478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.591403 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-kube-api-access-dcd7x" pod "45584a2a-6fd4-45fe-8b86-6add93774c95" (UID: "45584a2a-6fd4-45fe-8b86-6add93774c95"). InnerVolumeSpecName "kube-api-access-dcd7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:44.691070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.691041 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.691070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.691065 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.691070 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.691075 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.691290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.691084 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcd7x\" (UniqueName: \"kubernetes.io/projected/45584a2a-6fd4-45fe-8b86-6add93774c95-kube-api-access-dcd7x\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:44.835645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.835571 4294 generic.go:356] "Generic (PLEG): container finished" podID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerID="eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52" exitCode=0 Oct 31 22:18:44.835645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.835610 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" event={"ID":"45584a2a-6fd4-45fe-8b86-6add93774c95","Type":"ContainerDied","Data":"eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52"} Oct 31 22:18:44.835645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.835632 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" Oct 31 22:18:44.835912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.835653 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb" event={"ID":"45584a2a-6fd4-45fe-8b86-6add93774c95","Type":"ContainerDied","Data":"5594045b8645dec1924b016cc0d8359e102dbc81f140a52ecb39694eac6e1e52"} Oct 31 22:18:44.835912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.835673 4294 scope.go:122] "RemoveContainer" containerID="eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52" Oct 31 22:18:44.846660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.846642 4294 scope.go:122] "RemoveContainer" containerID="9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41" Oct 31 22:18:44.856108 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.856082 4294 scope.go:122] "RemoveContainer" containerID="eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52" Oct 31 22:18:44.856394 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:44.856376 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52\": container with ID starting with eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52 not found: ID does not exist" containerID="eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52" Oct 31 22:18:44.856465 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.856401 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52"} err="failed to get container status \"eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52\": rpc error: code = NotFound desc = could not find container \"eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52\": container with ID starting with eeffe35dc9ba93e4204c23c03ad8d5551eb2ff8d5280efdf694d36e5e3491d52 not found: ID does not exist" Oct 31 22:18:44.856465 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.856418 4294 scope.go:122] "RemoveContainer" containerID="9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41" Oct 31 22:18:44.856693 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:44.856672 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41\": container with ID starting with 9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41 not found: ID does not exist" containerID="9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41" Oct 31 22:18:44.856743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.856704 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41"} err="failed to get container status \"9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41\": rpc error: code = NotFound desc = could not find container \"9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41\": container with ID starting with 9f140dbcf0819bff90419d44deaa47f613a773d58678510b8386ffcc43109d41 not found: ID does not exist" Oct 31 22:18:44.891596 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.891573 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb"] Oct 31 22:18:44.904464 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:44.904441 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-696f9a-openshift-default-65db49bc97-mvhzb"] Oct 31 22:18:45.984294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:45.984261 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" path="/var/lib/kubelet/pods/45584a2a-6fd4-45fe-8b86-6add93774c95/volumes" Oct 31 22:18:46.961255 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:46.961214 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" podUID="cff86642-c360-4e03-a883-11efd7d6fdfe" containerName="main" containerID="cri-o://f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558" gracePeriod=45 Oct 31 22:18:47.219499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.219439 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:18:47.322809 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.322781 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-home\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-home\") pod \"cff86642-c360-4e03-a883-11efd7d6fdfe\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " Oct 31 22:18:47.322961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.322828 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-model-cache\") pod \"cff86642-c360-4e03-a883-11efd7d6fdfe\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " Oct 31 22:18:47.322961 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.322934 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-tmp-dir\") pod \"cff86642-c360-4e03-a883-11efd7d6fdfe\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " Oct 31 22:18:47.323039 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323027 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-dshm\") pod \"cff86642-c360-4e03-a883-11efd7d6fdfe\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " Oct 31 22:18:47.323100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323072 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-home" pod "cff86642-c360-4e03-a883-11efd7d6fdfe" (UID: "cff86642-c360-4e03-a883-11efd7d6fdfe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:47.323100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323081 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cff86642-c360-4e03-a883-11efd7d6fdfe-tls-certs\" (UniqueName: \"kubernetes.io/secret/cff86642-c360-4e03-a883-11efd7d6fdfe-tls-certs\") pod \"cff86642-c360-4e03-a883-11efd7d6fdfe\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " Oct 31 22:18:47.323188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323124 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-model-cache" pod "cff86642-c360-4e03-a883-11efd7d6fdfe" (UID: "cff86642-c360-4e03-a883-11efd7d6fdfe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:47.323222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323195 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff86642-c360-4e03-a883-11efd7d6fdfe-kube-api-access-4swqx\" (UniqueName: \"kubernetes.io/projected/cff86642-c360-4e03-a883-11efd7d6fdfe-kube-api-access-4swqx\") pod \"cff86642-c360-4e03-a883-11efd7d6fdfe\" (UID: \"cff86642-c360-4e03-a883-11efd7d6fdfe\") " Oct 31 22:18:47.323267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323225 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-tmp-dir" pod "cff86642-c360-4e03-a883-11efd7d6fdfe" (UID: "cff86642-c360-4e03-a883-11efd7d6fdfe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:47.323616 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323596 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:47.323616 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323617 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:47.323805 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.323652 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:47.325372 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.325354 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff86642-c360-4e03-a883-11efd7d6fdfe-kube-api-access-4swqx" pod "cff86642-c360-4e03-a883-11efd7d6fdfe" (UID: "cff86642-c360-4e03-a883-11efd7d6fdfe"). InnerVolumeSpecName "kube-api-access-4swqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:18:47.325580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.325557 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cff86642-c360-4e03-a883-11efd7d6fdfe-tls-certs" pod "cff86642-c360-4e03-a883-11efd7d6fdfe" (UID: "cff86642-c360-4e03-a883-11efd7d6fdfe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:18:47.325580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.325564 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-dshm" pod "cff86642-c360-4e03-a883-11efd7d6fdfe" (UID: "cff86642-c360-4e03-a883-11efd7d6fdfe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:18:47.425192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.425169 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cff86642-c360-4e03-a883-11efd7d6fdfe-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:47.425192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.425191 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cff86642-c360-4e03-a883-11efd7d6fdfe-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:47.425364 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.425201 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4swqx\" (UniqueName: \"kubernetes.io/projected/cff86642-c360-4e03-a883-11efd7d6fdfe-kube-api-access-4swqx\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:18:47.857366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.857332 4294 generic.go:356] "Generic (PLEG): container finished" podID="cff86642-c360-4e03-a883-11efd7d6fdfe" containerID="f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558" exitCode=0 Oct 31 22:18:47.857366 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.857364 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" Oct 31 22:18:47.857627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.857383 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" event={"ID":"cff86642-c360-4e03-a883-11efd7d6fdfe","Type":"ContainerDied","Data":"f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558"} Oct 31 22:18:47.857627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.857416 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" event={"ID":"cff86642-c360-4e03-a883-11efd7d6fdfe","Type":"ContainerDied","Data":"0199c863d64bf5c306458ff2303d15e25b089c9d2db78405d6e845a089f5b6ec"} Oct 31 22:18:47.857627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.857432 4294 scope.go:122] "RemoveContainer" containerID="f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558" Oct 31 22:18:47.867481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.867462 4294 scope.go:122] "RemoveContainer" containerID="f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558" Oct 31 22:18:47.867737 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:18:47.867721 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558\": container with ID starting with f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558 not found: ID does not exist" containerID="f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558" Oct 31 22:18:47.867789 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.867745 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558"} err="failed to get container status \"f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558\": rpc error: code = NotFound desc = could not find container \"f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558\": container with ID starting with f257343f3c0b4c2e0f62dd01c8269dace086155b41d42a1e19ebf315781a8558 not found: ID does not exist" Oct 31 22:18:47.912411 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.912389 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs"] Oct 31 22:18:47.927911 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.927884 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs"] Oct 31 22:18:47.984010 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:47.983981 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff86642-c360-4e03-a883-11efd7d6fdfe" path="/var/lib/kubelet/pods/cff86642-c360-4e03-a883-11efd7d6fdfe/volumes" Oct 31 22:18:48.214563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:48.214475 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-5c54fc/test-model-5c54fc-kserve-bd9466695-vkdcs" podUID="cff86642-c360-4e03-a883-11efd7d6fdfe" containerName="main" probeResult="failure" output="Get \"https://10.134.1.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 22:18:49.913722 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:49.913683 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" podUID="835a2433-6d43-4795-b33f-3a8f3bbf586d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:49.914116 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:49.913863 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" podUID="97f1ea75-483a-44a8-aab9-ce156938abe5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:50.719347 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:50.719316 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l"] Oct 31 22:18:50.731574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:50.731547 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c744c0-fks5l"] Oct 31 22:18:51.983724 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:51.983693 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2f9f3f-6a02-4f65-be06-f62bcf092644" path="/var/lib/kubelet/pods/6e2f9f3f-6a02-4f65-be06-f62bcf092644/volumes" Oct 31 22:18:56.100243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:56.100210 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5"] Oct 31 22:18:56.112727 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:56.112698 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0cce-vrwc5"] Oct 31 22:18:56.697816 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:56.697782 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889"] Oct 31 22:18:56.712045 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:56.712013 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4a73b1-rv889"] Oct 31 22:18:57.983750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:57.983717 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0afe9610-ed31-4a99-8a83-f87ab0abb694" path="/var/lib/kubelet/pods/0afe9610-ed31-4a99-8a83-f87ab0abb694/volumes" Oct 31 22:18:57.984167 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:57.984152 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edbca627-1eb9-4242-baee-9817a22473e2" path="/var/lib/kubelet/pods/edbca627-1eb9-4242-baee-9817a22473e2/volumes" Oct 31 22:18:58.855332 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855284 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k"] Oct 31 22:18:58.855799 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855779 4294 cpu_manager.go:414] "containerMap: removing container" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" Oct 31 22:18:58.855900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855799 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" Oct 31 22:18:58.855900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855812 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0afe9610-ed31-4a99-8a83-f87ab0abb694" containerName="revoke-keys" Oct 31 22:18:58.855900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855819 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0afe9610-ed31-4a99-8a83-f87ab0abb694" containerName="revoke-keys" Oct 31 22:18:58.855900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855859 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cff86642-c360-4e03-a883-11efd7d6fdfe" containerName="main" Oct 31 22:18:58.855900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855867 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff86642-c360-4e03-a883-11efd7d6fdfe" containerName="main" Oct 31 22:18:58.856157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.855992 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" Oct 31 22:18:58.856157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.856005 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff86642-c360-4e03-a883-11efd7d6fdfe" containerName="main" Oct 31 22:18:58.856157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.856023 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0afe9610-ed31-4a99-8a83-f87ab0abb694" containerName="revoke-keys" Oct 31 22:18:58.863336 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.862853 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:58.878363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.878338 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k"] Oct 31 22:18:58.888806 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.888784 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:18:58.888926 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.888865 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:18:58.971462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.971434 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r579\" (UniqueName: \"kubernetes.io/projected/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-kube-api-access-6r579\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:58.971620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.971596 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:58.971668 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:58.971657 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.072768 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.072740 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r579\" (UniqueName: \"kubernetes.io/projected/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-kube-api-access-6r579\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.073084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.072853 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.073084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.072888 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.073538 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.073514 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.075100 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.075079 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.097309 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.097286 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r579\" (UniqueName: \"kubernetes.io/projected/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-kube-api-access-6r579\") pod \"maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.178427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.178368 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:18:59.475189 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.475156 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k"] Oct 31 22:18:59.476867 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:18:59.476839 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c793834_43c8_4ce0_bbe2_09bfa4c1c482.slice/crio-40f2ef29d626a2b8708819e1c314dc03141209ffd53859160cd833d13f539cf2 WatchSource:0}: Error finding container 40f2ef29d626a2b8708819e1c314dc03141209ffd53859160cd833d13f539cf2: Status 404 returned error can't find the container with id 40f2ef29d626a2b8708819e1c314dc03141209ffd53859160cd833d13f539cf2 Oct 31 22:18:59.913228 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.913193 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" podUID="97f1ea75-483a-44a8-aab9-ce156938abe5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:59.913413 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.913201 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" podUID="835a2433-6d43-4795-b33f-3a8f3bbf586d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:18:59.942334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.942290 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" event={"ID":"4c793834-43c8-4ce0-bbe2-09bfa4c1c482","Type":"ContainerStarted","Data":"72143a4ff526e17c02004633e6e564d02c3d29b8f4db02923e20fefb3daed549"} Oct 31 22:18:59.942495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.942361 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" event={"ID":"4c793834-43c8-4ce0-bbe2-09bfa4c1c482","Type":"ContainerStarted","Data":"40f2ef29d626a2b8708819e1c314dc03141209ffd53859160cd833d13f539cf2"} Oct 31 22:18:59.986819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:18:59.986770 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" podStartSLOduration=1.986754043 podStartE2EDuration="1.986754043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:18:59.984530158 +0000 UTC m=+3780.109637387" watchObservedRunningTime="2026-10-31 22:18:59.986754043 +0000 UTC m=+3780.111861277" Oct 31 22:19:00.843788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.843763 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 22:19:00.877151 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.877130 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 22:19:00.950277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.950244 4294 generic.go:356] "Generic (PLEG): container finished" podID="4c793834-43c8-4ce0-bbe2-09bfa4c1c482" containerID="72143a4ff526e17c02004633e6e564d02c3d29b8f4db02923e20fefb3daed549" exitCode=0 Oct 31 22:19:00.950465 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.950351 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" event={"ID":"4c793834-43c8-4ce0-bbe2-09bfa4c1c482","Type":"ContainerDied","Data":"72143a4ff526e17c02004633e6e564d02c3d29b8f4db02923e20fefb3daed549"} Oct 31 22:19:00.951670 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.951646 4294 generic.go:356] "Generic (PLEG): container finished" podID="97f1ea75-483a-44a8-aab9-ce156938abe5" containerID="2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd" exitCode=0 Oct 31 22:19:00.951791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.951691 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" Oct 31 22:19:00.951791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.951733 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" event={"ID":"97f1ea75-483a-44a8-aab9-ce156938abe5","Type":"ContainerDied","Data":"2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd"} Oct 31 22:19:00.951791 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.951780 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6" event={"ID":"97f1ea75-483a-44a8-aab9-ce156938abe5","Type":"ContainerDied","Data":"0fb22bd15e698b6378cf62001ff2eaf2f3ef8c980a25ffd190200888398b8f91"} Oct 31 22:19:00.951964 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.951809 4294 scope.go:122] "RemoveContainer" containerID="2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd" Oct 31 22:19:00.953147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.953127 4294 generic.go:356] "Generic (PLEG): container finished" podID="835a2433-6d43-4795-b33f-3a8f3bbf586d" containerID="2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258" exitCode=0 Oct 31 22:19:00.953267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.953168 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" event={"ID":"835a2433-6d43-4795-b33f-3a8f3bbf586d","Type":"ContainerDied","Data":"2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258"} Oct 31 22:19:00.953267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.953173 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" Oct 31 22:19:00.953267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.953198 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6" event={"ID":"835a2433-6d43-4795-b33f-3a8f3bbf586d","Type":"ContainerDied","Data":"8f7c9cd2df1d7a1f08e23120113bb80bd8e6dd6c35775a8194440172f4be2382"} Oct 31 22:19:00.964437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.964417 4294 scope.go:122] "RemoveContainer" containerID="2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd" Oct 31 22:19:00.964743 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:00.964710 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd\": container with ID starting with 2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd not found: ID does not exist" containerID="2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd" Oct 31 22:19:00.964833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.964749 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd"} err="failed to get container status \"2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd\": rpc error: code = NotFound desc = could not find container \"2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd\": container with ID starting with 2191044b531d7430745c545d9fca328ffccb44d21edf70d8e7efe4c2d2b92ecd not found: ID does not exist" Oct 31 22:19:00.964833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.964770 4294 scope.go:122] "RemoveContainer" containerID="2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258" Oct 31 22:19:00.974356 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.974339 4294 scope.go:122] "RemoveContainer" containerID="2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258" Oct 31 22:19:00.974622 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:00.974603 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258\": container with ID starting with 2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258 not found: ID does not exist" containerID="2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258" Oct 31 22:19:00.974681 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.974632 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258"} err="failed to get container status \"2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258\": rpc error: code = NotFound desc = could not find container \"2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258\": container with ID starting with 2d385d0ed96edeae5099b13faecc4c2db2a3ea03250c1ee3086d2bd9e3e1b258 not found: ID does not exist" Oct 31 22:19:00.993250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.993226 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f1ea75-483a-44a8-aab9-ce156938abe5-kube-api-access-nlspk\" (UniqueName: \"kubernetes.io/projected/97f1ea75-483a-44a8-aab9-ce156938abe5-kube-api-access-nlspk\") pod \"97f1ea75-483a-44a8-aab9-ce156938abe5\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " Oct 31 22:19:00.993432 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.993353 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f1ea75-483a-44a8-aab9-ce156938abe5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f1ea75-483a-44a8-aab9-ce156938abe5-plugins-config-volume\") pod \"97f1ea75-483a-44a8-aab9-ce156938abe5\" (UID: \"97f1ea75-483a-44a8-aab9-ce156938abe5\") " Oct 31 22:19:00.993505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.993458 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835a2433-6d43-4795-b33f-3a8f3bbf586d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835a2433-6d43-4795-b33f-3a8f3bbf586d-plugins-config-volume\") pod \"835a2433-6d43-4795-b33f-3a8f3bbf586d\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " Oct 31 22:19:00.993562 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.993500 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835a2433-6d43-4795-b33f-3a8f3bbf586d-kube-api-access-8wx49\" (UniqueName: \"kubernetes.io/projected/835a2433-6d43-4795-b33f-3a8f3bbf586d-kube-api-access-8wx49\") pod \"835a2433-6d43-4795-b33f-3a8f3bbf586d\" (UID: \"835a2433-6d43-4795-b33f-3a8f3bbf586d\") " Oct 31 22:19:00.994137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.994113 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835a2433-6d43-4795-b33f-3a8f3bbf586d-plugins-config-volume" pod "835a2433-6d43-4795-b33f-3a8f3bbf586d" (UID: "835a2433-6d43-4795-b33f-3a8f3bbf586d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:00.994394 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.994373 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f1ea75-483a-44a8-aab9-ce156938abe5-plugins-config-volume" pod "97f1ea75-483a-44a8-aab9-ce156938abe5" (UID: "97f1ea75-483a-44a8-aab9-ce156938abe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:00.995377 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.995356 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f1ea75-483a-44a8-aab9-ce156938abe5-kube-api-access-nlspk" pod "97f1ea75-483a-44a8-aab9-ce156938abe5" (UID: "97f1ea75-483a-44a8-aab9-ce156938abe5"). InnerVolumeSpecName "kube-api-access-nlspk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:00.995576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:00.995557 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835a2433-6d43-4795-b33f-3a8f3bbf586d-kube-api-access-8wx49" pod "835a2433-6d43-4795-b33f-3a8f3bbf586d" (UID: "835a2433-6d43-4795-b33f-3a8f3bbf586d"). InnerVolumeSpecName "kube-api-access-8wx49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:01.094749 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.094719 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/835a2433-6d43-4795-b33f-3a8f3bbf586d-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:01.094749 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.094746 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wx49\" (UniqueName: \"kubernetes.io/projected/835a2433-6d43-4795-b33f-3a8f3bbf586d-kube-api-access-8wx49\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:01.094901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.094762 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlspk\" (UniqueName: \"kubernetes.io/projected/97f1ea75-483a-44a8-aab9-ce156938abe5-kube-api-access-nlspk\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:01.094901 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.094776 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97f1ea75-483a-44a8-aab9-ce156938abe5-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:01.300551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.300507 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6"] Oct 31 22:19:01.305386 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.305360 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-c744c0-7c8c44f8c-5lcg6"] Oct 31 22:19:01.310683 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:01.310657 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:19:01.342747 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.342694 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6"] Oct 31 22:19:01.345548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.345526 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-c744c0-7d8d5544b9-n7kw6"] Oct 31 22:19:01.983202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.983166 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835a2433-6d43-4795-b33f-3a8f3bbf586d" path="/var/lib/kubelet/pods/835a2433-6d43-4795-b33f-3a8f3bbf586d/volumes" Oct 31 22:19:01.983786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:01.983764 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f1ea75-483a-44a8-aab9-ce156938abe5" path="/var/lib/kubelet/pods/97f1ea75-483a-44a8-aab9-ce156938abe5/volumes" Oct 31 22:19:02.100044 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.100023 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:19:02.206537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.206511 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-openshift-ca\") pod \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " Oct 31 22:19:02.206716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.206606 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-xks-ca\") pod \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " Oct 31 22:19:02.206716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.206685 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-kube-api-access-6r579\" (UniqueName: \"kubernetes.io/projected/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-kube-api-access-6r579\") pod \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\" (UID: \"4c793834-43c8-4ce0-bbe2-09bfa4c1c482\") " Oct 31 22:19:02.206933 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.206909 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-openshift-ca" pod "4c793834-43c8-4ce0-bbe2-09bfa4c1c482" (UID: "4c793834-43c8-4ce0-bbe2-09bfa4c1c482"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:02.207019 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.206996 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:02.208731 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.208707 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-kube-api-access-6r579" pod "4c793834-43c8-4ce0-bbe2-09bfa4c1c482" (UID: "4c793834-43c8-4ce0-bbe2-09bfa4c1c482"). InnerVolumeSpecName "kube-api-access-6r579". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:02.208808 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.208709 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-xks-ca" pod "4c793834-43c8-4ce0-bbe2-09bfa4c1c482" (UID: "4c793834-43c8-4ce0-bbe2-09bfa4c1c482"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:02.307548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.307523 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:02.307548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.307547 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r579\" (UniqueName: \"kubernetes.io/projected/4c793834-43c8-4ce0-bbe2-09bfa4c1c482-kube-api-access-6r579\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:02.970534 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.970501 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" Oct 31 22:19:02.970742 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.970501 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k" event={"ID":"4c793834-43c8-4ce0-bbe2-09bfa4c1c482","Type":"ContainerDied","Data":"40f2ef29d626a2b8708819e1c314dc03141209ffd53859160cd833d13f539cf2"} Oct 31 22:19:02.970742 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:02.970655 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40f2ef29d626a2b8708819e1c314dc03141209ffd53859160cd833d13f539cf2" Oct 31 22:19:03.602977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:03.602940 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf"] Oct 31 22:19:03.919913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:03.919835 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8"] Oct 31 22:19:03.920084 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:03.920059 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" containerID="cri-o://3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6" gracePeriod=30 Oct 31 22:19:04.205922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.205843 4294 scope.go:122] "RemoveContainer" containerID="20e82cbf45467b85a5c08db65677ea9fe0df42e4c188ad16bfa00b940afe4cf8" Oct 31 22:19:04.215998 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.215974 4294 scope.go:122] "RemoveContainer" containerID="88432bd681c6b2a290a8456d2e16290f514a1e6b4a9590a83d267aee9bd369c6" Oct 31 22:19:04.503956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.503915 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6"] Oct 31 22:19:04.504447 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504422 4294 cpu_manager.go:414] "containerMap: removing container" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" Oct 31 22:19:04.504447 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504440 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" Oct 31 22:19:04.504557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504469 4294 cpu_manager.go:414] "containerMap: removing container" podUID="835a2433-6d43-4795-b33f-3a8f3bbf586d" containerName="payload-processing" Oct 31 22:19:04.504557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504475 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835a2433-6d43-4795-b33f-3a8f3bbf586d" containerName="payload-processing" Oct 31 22:19:04.504557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504491 4294 cpu_manager.go:414] "containerMap: removing container" podUID="97f1ea75-483a-44a8-aab9-ce156938abe5" containerName="payload-pre-processing" Oct 31 22:19:04.504557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504498 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f1ea75-483a-44a8-aab9-ce156938abe5" containerName="payload-pre-processing" Oct 31 22:19:04.504557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504508 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4c793834-43c8-4ce0-bbe2-09bfa4c1c482" containerName="revoke-keys" Oct 31 22:19:04.504557 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504513 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c793834-43c8-4ce0-bbe2-09bfa4c1c482" containerName="revoke-keys" Oct 31 22:19:04.504759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504586 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f1ea75-483a-44a8-aab9-ce156938abe5" containerName="payload-pre-processing" Oct 31 22:19:04.504759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504598 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c793834-43c8-4ce0-bbe2-09bfa4c1c482" containerName="revoke-keys" Oct 31 22:19:04.504759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504609 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="835a2433-6d43-4795-b33f-3a8f3bbf586d" containerName="payload-processing" Oct 31 22:19:04.504759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.504615 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="45584a2a-6fd4-45fe-8b86-6add93774c95" containerName="istio-proxy" Oct 31 22:19:04.509665 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.509647 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.527199 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.527175 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6"] Oct 31 22:19:04.537116 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.537095 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:19:04.537412 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.537393 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:19:04.548040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.548019 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.548169 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.548153 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45shm\" (UniqueName: \"kubernetes.io/projected/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-kube-api-access-45shm\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.548251 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.548232 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.648970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.648932 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.649488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.649003 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45shm\" (UniqueName: \"kubernetes.io/projected/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-kube-api-access-45shm\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.649488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.649034 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.649734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.649714 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.653778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.653748 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.674343 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.674272 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45shm\" (UniqueName: \"kubernetes.io/projected/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-kube-api-access-45shm\") pod \"maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:04.695243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.695205 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j"] Oct 31 22:19:04.696541 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.696508 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" containerID="cri-o://a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4" gracePeriod=30 Oct 31 22:19:04.829373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:04.828507 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:05.070947 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.070913 4294 patch_prober.go:29] interesting pod/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 22:19:05.071119 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.070969 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 22:19:05.140862 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.140834 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6"] Oct 31 22:19:05.142853 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:05.142826 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab0a982_6c4f_4d6e_9853_ee3ddfab28af.slice/crio-85e98d6ddd4194b32558b1fc07b1fee508eba62c5cb30ac0784ee9821dbbf76d WatchSource:0}: Error finding container 85e98d6ddd4194b32558b1fc07b1fee508eba62c5cb30ac0784ee9821dbbf76d: Status 404 returned error can't find the container with id 85e98d6ddd4194b32558b1fc07b1fee508eba62c5cb30ac0784ee9821dbbf76d Oct 31 22:19:05.280821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.280790 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj"] Oct 31 22:19:05.285445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.285416 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.303331 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.303294 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj"] Oct 31 22:19:05.357567 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.357538 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.357683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.357614 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjpg2\" (UniqueName: \"kubernetes.io/projected/502f4be6-ec90-4e19-b286-ab45137faf8a-kube-api-access-kjpg2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.357723 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.357698 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.458874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.458819 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.459005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.458893 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjpg2\" (UniqueName: \"kubernetes.io/projected/502f4be6-ec90-4e19-b286-ab45137faf8a-kube-api-access-kjpg2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.459005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.458927 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.459567 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.459545 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.461109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.461087 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.483026 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.483007 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjpg2\" (UniqueName: \"kubernetes.io/projected/502f4be6-ec90-4e19-b286-ab45137faf8a-kube-api-access-kjpg2\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.597683 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.597657 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:05.893759 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.893731 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj"] Oct 31 22:19:05.895679 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:05.895652 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod502f4be6_ec90_4e19_b286_ab45137faf8a.slice/crio-da76c80c1058842fe398723b7c3a37f7b1f93ab7eeba04ae08689c39d9f6118f WatchSource:0}: Error finding container da76c80c1058842fe398723b7c3a37f7b1f93ab7eeba04ae08689c39d9f6118f: Status 404 returned error can't find the container with id da76c80c1058842fe398723b7c3a37f7b1f93ab7eeba04ae08689c39d9f6118f Oct 31 22:19:05.991186 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.991155 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" event={"ID":"cab0a982-6c4f-4d6e-9853-ee3ddfab28af","Type":"ContainerStarted","Data":"981f6f22ef33b954e9521878eaadcabab971416c2e103d82d54d8adca988cc43"} Oct 31 22:19:05.991340 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.991215 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" event={"ID":"cab0a982-6c4f-4d6e-9853-ee3ddfab28af","Type":"ContainerStarted","Data":"85e98d6ddd4194b32558b1fc07b1fee508eba62c5cb30ac0784ee9821dbbf76d"} Oct 31 22:19:05.992445 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:05.992424 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" event={"ID":"502f4be6-ec90-4e19-b286-ab45137faf8a","Type":"ContainerStarted","Data":"da76c80c1058842fe398723b7c3a37f7b1f93ab7eeba04ae08689c39d9f6118f"} Oct 31 22:19:06.035651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:06.035612 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" podStartSLOduration=2.035600128 podStartE2EDuration="2.035600128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:19:06.032926987 +0000 UTC m=+3786.158034218" watchObservedRunningTime="2026-10-31 22:19:06.035600128 +0000 UTC m=+3786.160707378" Oct 31 22:19:06.999608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:06.999575 4294 generic.go:356] "Generic (PLEG): container finished" podID="cab0a982-6c4f-4d6e-9853-ee3ddfab28af" containerID="981f6f22ef33b954e9521878eaadcabab971416c2e103d82d54d8adca988cc43" exitCode=0 Oct 31 22:19:07.000033 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:06.999655 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" event={"ID":"cab0a982-6c4f-4d6e-9853-ee3ddfab28af","Type":"ContainerDied","Data":"981f6f22ef33b954e9521878eaadcabab971416c2e103d82d54d8adca988cc43"} Oct 31 22:19:07.001162 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:07.001139 4294 generic.go:356] "Generic (PLEG): container finished" podID="502f4be6-ec90-4e19-b286-ab45137faf8a" containerID="4e207c00caaf556ef4d57480577bc23732ea9856188cb259629c35674ae6417e" exitCode=0 Oct 31 22:19:07.001268 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:07.001174 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" event={"ID":"502f4be6-ec90-4e19-b286-ab45137faf8a","Type":"ContainerDied","Data":"4e207c00caaf556ef4d57480577bc23732ea9856188cb259629c35674ae6417e"} Oct 31 22:19:08.146441 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.146416 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:08.183960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.183930 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:08.195778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.195402 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-xks-ca\") pod \"502f4be6-ec90-4e19-b286-ab45137faf8a\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " Oct 31 22:19:08.195778 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.195460 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-kube-api-access-45shm\" (UniqueName: \"kubernetes.io/projected/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-kube-api-access-45shm\") pod \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " Oct 31 22:19:08.196093 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.196065 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/502f4be6-ec90-4e19-b286-ab45137faf8a-kube-api-access-kjpg2\" (UniqueName: \"kubernetes.io/projected/502f4be6-ec90-4e19-b286-ab45137faf8a-kube-api-access-kjpg2\") pod \"502f4be6-ec90-4e19-b286-ab45137faf8a\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " Oct 31 22:19:08.196274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.196136 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-xks-ca\") pod \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " Oct 31 22:19:08.196274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.196178 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-openshift-ca\") pod \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\" (UID: \"cab0a982-6c4f-4d6e-9853-ee3ddfab28af\") " Oct 31 22:19:08.196274 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.196255 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-openshift-ca\") pod \"502f4be6-ec90-4e19-b286-ab45137faf8a\" (UID: \"502f4be6-ec90-4e19-b286-ab45137faf8a\") " Oct 31 22:19:08.197225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.197008 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-openshift-ca" pod "502f4be6-ec90-4e19-b286-ab45137faf8a" (UID: "502f4be6-ec90-4e19-b286-ab45137faf8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:08.197728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.197706 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-openshift-ca" pod "cab0a982-6c4f-4d6e-9853-ee3ddfab28af" (UID: "cab0a982-6c4f-4d6e-9853-ee3ddfab28af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:08.197930 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.197898 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-xks-ca" pod "502f4be6-ec90-4e19-b286-ab45137faf8a" (UID: "502f4be6-ec90-4e19-b286-ab45137faf8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:08.198537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.198513 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-kube-api-access-45shm" pod "cab0a982-6c4f-4d6e-9853-ee3ddfab28af" (UID: "cab0a982-6c4f-4d6e-9853-ee3ddfab28af"). InnerVolumeSpecName "kube-api-access-45shm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:08.199618 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.199576 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/502f4be6-ec90-4e19-b286-ab45137faf8a-kube-api-access-kjpg2" pod "502f4be6-ec90-4e19-b286-ab45137faf8a" (UID: "502f4be6-ec90-4e19-b286-ab45137faf8a"). InnerVolumeSpecName "kube-api-access-kjpg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:08.199971 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.199951 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-xks-ca" pod "cab0a982-6c4f-4d6e-9853-ee3ddfab28af" (UID: "cab0a982-6c4f-4d6e-9853-ee3ddfab28af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:08.297985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.297905 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:08.297985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.297933 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45shm\" (UniqueName: \"kubernetes.io/projected/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-kube-api-access-45shm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:08.297985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.297944 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjpg2\" (UniqueName: \"kubernetes.io/projected/502f4be6-ec90-4e19-b286-ab45137faf8a-kube-api-access-kjpg2\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:08.297985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.297952 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:08.297985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.297960 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab0a982-6c4f-4d6e-9853-ee3ddfab28af-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:08.297985 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:08.297968 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/502f4be6-ec90-4e19-b286-ab45137faf8a-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.015214 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.015187 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" Oct 31 22:19:09.015378 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.015189 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6" event={"ID":"cab0a982-6c4f-4d6e-9853-ee3ddfab28af","Type":"ContainerDied","Data":"85e98d6ddd4194b32558b1fc07b1fee508eba62c5cb30ac0784ee9821dbbf76d"} Oct 31 22:19:09.015378 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.015347 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e98d6ddd4194b32558b1fc07b1fee508eba62c5cb30ac0784ee9821dbbf76d" Oct 31 22:19:09.016655 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.016636 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" Oct 31 22:19:09.016764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.016676 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj" event={"ID":"502f4be6-ec90-4e19-b286-ab45137faf8a","Type":"ContainerDied","Data":"da76c80c1058842fe398723b7c3a37f7b1f93ab7eeba04ae08689c39d9f6118f"} Oct 31 22:19:09.016764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.016716 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da76c80c1058842fe398723b7c3a37f7b1f93ab7eeba04ae08689c39d9f6118f" Oct 31 22:19:09.186609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.186554 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:19:09.206630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206607 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-socket\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206654 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-data\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206681 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-kube-api-access-mkc9h\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-kube-api-access-mkc9h\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206716 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-envoy\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206745 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206736 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-credential-socket\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206753 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01b1fb25-a674-4ce4-917c-964978046bdb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b1fb25-a674-4ce4-917c-964978046bdb-istio-podinfo\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206799 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b1fb25-a674-4ce4-917c-964978046bdb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b1fb25-a674-4ce4-917c-964978046bdb-istiod-ca-cert\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206832 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-certs\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.206968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206884 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-socket" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:09.206968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.206902 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-istio-token\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-istio-token\") pod \"01b1fb25-a674-4ce4-917c-964978046bdb\" (UID: \"01b1fb25-a674-4ce4-917c-964978046bdb\") " Oct 31 22:19:09.207226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207039 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-credential-socket" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:09.207226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207056 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-data" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:09.207449 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207169 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b1fb25-a674-4ce4-917c-964978046bdb-istiod-ca-cert" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:09.207449 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207248 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-certs" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:09.207532 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207520 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.207565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207536 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b1fb25-a674-4ce4-917c-964978046bdb-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.207565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207546 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.207565 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207557 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.207673 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.207567 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.208880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.208854 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-istio-token" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:09.209126 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.209109 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-envoy" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:09.209321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.209284 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01b1fb25-a674-4ce4-917c-964978046bdb-istio-podinfo" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:19:09.209391 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.209326 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-kube-api-access-mkc9h" pod "01b1fb25-a674-4ce4-917c-964978046bdb" (UID: "01b1fb25-a674-4ce4-917c-964978046bdb"). InnerVolumeSpecName "kube-api-access-mkc9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:09.308586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.308560 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkc9h\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-kube-api-access-mkc9h\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.308586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.308583 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b1fb25-a674-4ce4-917c-964978046bdb-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.308586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.308592 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b1fb25-a674-4ce4-917c-964978046bdb-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.308783 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.308599 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b1fb25-a674-4ce4-917c-964978046bdb-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:09.939608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:09.939581 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 22:19:10.017036 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017007 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-istio-token\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-istio-token\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017046 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-kube-api-access-qr4pd\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-kube-api-access-qr4pd\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017082 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-envoy\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017156 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-socket\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017206 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-certs\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017269 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-credential-socket\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017325 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/955053d6-46c4-4bc9-b339-c79676dc349f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955053d6-46c4-4bc9-b339-c79676dc349f-istio-podinfo\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017363 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-data\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.017609 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.017402 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/955053d6-46c4-4bc9-b339-c79676dc349f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955053d6-46c4-4bc9-b339-c79676dc349f-istiod-ca-cert\") pod \"955053d6-46c4-4bc9-b339-c79676dc349f\" (UID: \"955053d6-46c4-4bc9-b339-c79676dc349f\") " Oct 31 22:19:10.018086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.018057 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-credential-socket" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:10.018261 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.018242 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-certs" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:10.018370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.018325 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-socket" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:10.018437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.018398 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-data" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:10.018536 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.018511 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/955053d6-46c4-4bc9-b339-c79676dc349f-istiod-ca-cert" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:10.020351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.019705 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-kube-api-access-qr4pd" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "kube-api-access-qr4pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:10.020351 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.019775 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-istio-token" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:10.020660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.020631 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/955053d6-46c4-4bc9-b339-c79676dc349f-istio-podinfo" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:19:10.021881 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.021853 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-envoy" pod "955053d6-46c4-4bc9-b339-c79676dc349f" (UID: "955053d6-46c4-4bc9-b339-c79676dc349f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:10.025107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.025083 4294 generic.go:356] "Generic (PLEG): container finished" podID="955053d6-46c4-4bc9-b339-c79676dc349f" containerID="a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4" exitCode=0 Oct 31 22:19:10.025247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.025151 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" Oct 31 22:19:10.025247 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.025185 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" event={"ID":"955053d6-46c4-4bc9-b339-c79676dc349f","Type":"ContainerDied","Data":"a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4"} Oct 31 22:19:10.025442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.025250 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j" event={"ID":"955053d6-46c4-4bc9-b339-c79676dc349f","Type":"ContainerDied","Data":"c500f1a0ee7ae95403068e8c696aaae833e8bdaf355e2ab465ee029af2a30f36"} Oct 31 22:19:10.025442 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.025280 4294 scope.go:122] "RemoveContainer" containerID="a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4" Oct 31 22:19:10.031195 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.031173 4294 generic.go:356] "Generic (PLEG): container finished" podID="01b1fb25-a674-4ce4-917c-964978046bdb" containerID="3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6" exitCode=0 Oct 31 22:19:10.031314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.031226 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" Oct 31 22:19:10.031314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.031238 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" event={"ID":"01b1fb25-a674-4ce4-917c-964978046bdb","Type":"ContainerDied","Data":"3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6"} Oct 31 22:19:10.031314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.031281 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8" event={"ID":"01b1fb25-a674-4ce4-917c-964978046bdb","Type":"ContainerDied","Data":"18265c1dc5644904baf4b3be720ed701bc8e045dc23b4d9d6d9b9afef663ed01"} Oct 31 22:19:10.041008 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.040890 4294 scope.go:122] "RemoveContainer" containerID="0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a" Oct 31 22:19:10.051499 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.051479 4294 scope.go:122] "RemoveContainer" containerID="a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4" Oct 31 22:19:10.051767 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:10.051744 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4\": container with ID starting with a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4 not found: ID does not exist" containerID="a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4" Oct 31 22:19:10.051824 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.051777 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4"} err="failed to get container status \"a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4\": rpc error: code = NotFound desc = could not find container \"a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4\": container with ID starting with a617a37aac44091b035d8d6a1b69b89c5b67e90e4866fd5c3a461d033d6a36e4 not found: ID does not exist" Oct 31 22:19:10.051824 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.051794 4294 scope.go:122] "RemoveContainer" containerID="0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a" Oct 31 22:19:10.052029 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:10.052008 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a\": container with ID starting with 0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a not found: ID does not exist" containerID="0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a" Oct 31 22:19:10.052106 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.052040 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a"} err="failed to get container status \"0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a\": rpc error: code = NotFound desc = could not find container \"0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a\": container with ID starting with 0d169d6143cd89d627533bd99415fe2efcb9f204d07cef1288efd62cec9aab2a not found: ID does not exist" Oct 31 22:19:10.052106 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.052060 4294 scope.go:122] "RemoveContainer" containerID="3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6" Oct 31 22:19:10.111350 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.111314 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8"] Oct 31 22:19:10.112884 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.112858 4294 scope.go:122] "RemoveContainer" containerID="a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa" Oct 31 22:19:10.115123 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.115089 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0cce-openshift-default-585cd6bb59-snkm8"] Oct 31 22:19:10.119283 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119258 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119290 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/955053d6-46c4-4bc9-b339-c79676dc349f-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119346 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119360 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/955053d6-46c4-4bc9-b339-c79676dc349f-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119372 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119385 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr4pd\" (UniqueName: \"kubernetes.io/projected/955053d6-46c4-4bc9-b339-c79676dc349f-kube-api-access-qr4pd\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119399 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119412 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.119439 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.119424 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/955053d6-46c4-4bc9-b339-c79676dc349f-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:10.138680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.138659 4294 scope.go:122] "RemoveContainer" containerID="3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6" Oct 31 22:19:10.139022 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:10.139001 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6\": container with ID starting with 3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6 not found: ID does not exist" containerID="3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6" Oct 31 22:19:10.139114 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.139035 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6"} err="failed to get container status \"3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6\": rpc error: code = NotFound desc = could not find container \"3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6\": container with ID starting with 3f159d46bcc58ab5d4e2c169df6eb1fdc1e2eff0870f7ff291f9b44694b2d1d6 not found: ID does not exist" Oct 31 22:19:10.139114 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.139053 4294 scope.go:122] "RemoveContainer" containerID="a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa" Oct 31 22:19:10.139365 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:10.139348 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa\": container with ID starting with a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa not found: ID does not exist" containerID="a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa" Oct 31 22:19:10.139424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.139370 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa"} err="failed to get container status \"a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa\": rpc error: code = NotFound desc = could not find container \"a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa\": container with ID starting with a8d0024796ec32bf38f814c9bda182a451be5e829017687690ddcf4dfd0229fa not found: ID does not exist" Oct 31 22:19:10.151591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.151561 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j"] Oct 31 22:19:10.157505 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:10.157484 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4a73b1-openshift-default-5d7947885-jvj9j"] Oct 31 22:19:11.983594 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:11.983560 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" path="/var/lib/kubelet/pods/01b1fb25-a674-4ce4-917c-964978046bdb/volumes" Oct 31 22:19:11.984037 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:11.984023 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" path="/var/lib/kubelet/pods/955053d6-46c4-4bc9-b339-c79676dc349f/volumes" Oct 31 22:19:13.837266 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837233 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd"] Oct 31 22:19:13.837675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837650 4294 cpu_manager.go:414] "containerMap: removing container" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" Oct 31 22:19:13.837675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837661 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" Oct 31 22:19:13.837675 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837675 4294 cpu_manager.go:414] "containerMap: removing container" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837681 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837693 4294 cpu_manager.go:414] "containerMap: removing container" podUID="502f4be6-ec90-4e19-b286-ab45137faf8a" containerName="revoke-keys" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837699 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="502f4be6-ec90-4e19-b286-ab45137faf8a" containerName="revoke-keys" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837706 4294 cpu_manager.go:414] "containerMap: removing container" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837710 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837720 4294 cpu_manager.go:414] "containerMap: removing container" podUID="cab0a982-6c4f-4d6e-9853-ee3ddfab28af" containerName="revoke-keys" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837725 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab0a982-6c4f-4d6e-9853-ee3ddfab28af" containerName="revoke-keys" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837733 4294 cpu_manager.go:414] "containerMap: removing container" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" Oct 31 22:19:13.837785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837737 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" Oct 31 22:19:13.838064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837801 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" Oct 31 22:19:13.838064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837810 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b1fb25-a674-4ce4-917c-964978046bdb" containerName="istio-proxy" Oct 31 22:19:13.838064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837816 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="502f4be6-ec90-4e19-b286-ab45137faf8a" containerName="revoke-keys" Oct 31 22:19:13.838064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837821 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" Oct 31 22:19:13.838064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837825 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab0a982-6c4f-4d6e-9853-ee3ddfab28af" containerName="revoke-keys" Oct 31 22:19:13.838064 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.837831 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="955053d6-46c4-4bc9-b339-c79676dc349f" containerName="istio-proxy" Oct 31 22:19:13.842286 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.842267 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:13.859786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.859765 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5"] Oct 31 22:19:13.863845 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.863823 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:13.864871 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.864852 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd"] Oct 31 22:19:13.870749 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.869993 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:19:13.870749 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.870152 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:19:13.883011 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.882985 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5"] Oct 31 22:19:13.955734 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.955705 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:13.955889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.955743 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:13.955889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.955774 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:13.955889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.955842 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:13.955889 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.955861 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57zw\" (UniqueName: \"kubernetes.io/projected/8aac996a-26d0-4c19-b7e0-23ff356ebe37-kube-api-access-m57zw\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:13.956078 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:13.955986 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xskns\" (UniqueName: \"kubernetes.io/projected/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-kube-api-access-xskns\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.056989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.056960 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:14.057137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.057009 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.057137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.057039 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.057137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.057067 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57zw\" (UniqueName: \"kubernetes.io/projected/8aac996a-26d0-4c19-b7e0-23ff356ebe37-kube-api-access-m57zw\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:14.057380 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.057186 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xskns\" (UniqueName: \"kubernetes.io/projected/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-kube-api-access-xskns\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.057380 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.057248 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:14.057729 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.057709 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:14.057820 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.057798 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.059516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.059497 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:14.059598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.059514 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.082527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.082500 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57zw\" (UniqueName: \"kubernetes.io/projected/8aac996a-26d0-4c19-b7e0-23ff356ebe37-kube-api-access-m57zw\") pod \"maas-api-revoke-sub-961790d1-vjxh5\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:14.082614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.082586 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xskns\" (UniqueName: \"kubernetes.io/projected/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-kube-api-access-xskns\") pod \"maas-api-revoke-sub-8caf3a09-hxfkd\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.154977 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.154903 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:14.179903 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.179875 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:14.462658 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.462633 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd"] Oct 31 22:19:14.464399 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:14.464364 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9e5b6e_0f5e_4cbe_ba83_4e12ae4422b4.slice/crio-25f8ad60a659811b6b73e8fe80714765ee47f3c98440c86e17e74a1f33b3b581 WatchSource:0}: Error finding container 25f8ad60a659811b6b73e8fe80714765ee47f3c98440c86e17e74a1f33b3b581: Status 404 returned error can't find the container with id 25f8ad60a659811b6b73e8fe80714765ee47f3c98440c86e17e74a1f33b3b581 Oct 31 22:19:14.491056 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:14.491032 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5"] Oct 31 22:19:14.492746 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:14.492721 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aac996a_26d0_4c19_b7e0_23ff356ebe37.slice/crio-79729cb475e757ddb3cdca8c20b6519b3f193824ea0418faf99807c2fbca456d WatchSource:0}: Error finding container 79729cb475e757ddb3cdca8c20b6519b3f193824ea0418faf99807c2fbca456d: Status 404 returned error can't find the container with id 79729cb475e757ddb3cdca8c20b6519b3f193824ea0418faf99807c2fbca456d Oct 31 22:19:15.065688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:15.065591 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" event={"ID":"8aac996a-26d0-4c19-b7e0-23ff356ebe37","Type":"ContainerStarted","Data":"c25bf002c5dab4d5d51180fa10e344311021a11889fa5bed3cb2c4c4f3a1015e"} Oct 31 22:19:15.065688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:15.065656 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" event={"ID":"8aac996a-26d0-4c19-b7e0-23ff356ebe37","Type":"ContainerStarted","Data":"79729cb475e757ddb3cdca8c20b6519b3f193824ea0418faf99807c2fbca456d"} Oct 31 22:19:15.067527 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:15.067497 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" event={"ID":"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4","Type":"ContainerStarted","Data":"02800c3ea8238103db2490c048814e8970735c70c42e5004e1bd08bd56507333"} Oct 31 22:19:15.067650 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:15.067551 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" event={"ID":"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4","Type":"ContainerStarted","Data":"25f8ad60a659811b6b73e8fe80714765ee47f3c98440c86e17e74a1f33b3b581"} Oct 31 22:19:15.107627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:15.107578 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" podStartSLOduration=2.107563064 podStartE2EDuration="2.107563064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:19:15.107039251 +0000 UTC m=+3795.232146653" watchObservedRunningTime="2026-10-31 22:19:15.107563064 +0000 UTC m=+3795.232670294" Oct 31 22:19:15.148672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:15.148623 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" podStartSLOduration=2.14860701 podStartE2EDuration="2.14860701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:19:15.148105279 +0000 UTC m=+3795.273212510" watchObservedRunningTime="2026-10-31 22:19:15.14860701 +0000 UTC m=+3795.273714244" Oct 31 22:19:15.790348 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:15.789699 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg"] Oct 31 22:19:16.075435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:16.075340 4294 generic.go:356] "Generic (PLEG): container finished" podID="8aac996a-26d0-4c19-b7e0-23ff356ebe37" containerID="c25bf002c5dab4d5d51180fa10e344311021a11889fa5bed3cb2c4c4f3a1015e" exitCode=0 Oct 31 22:19:16.075435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:16.075417 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" event={"ID":"8aac996a-26d0-4c19-b7e0-23ff356ebe37","Type":"ContainerDied","Data":"c25bf002c5dab4d5d51180fa10e344311021a11889fa5bed3cb2c4c4f3a1015e"} Oct 31 22:19:16.076922 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:16.076894 4294 generic.go:356] "Generic (PLEG): container finished" podID="8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4" containerID="02800c3ea8238103db2490c048814e8970735c70c42e5004e1bd08bd56507333" exitCode=0 Oct 31 22:19:16.077052 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:16.076924 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" event={"ID":"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4","Type":"ContainerDied","Data":"02800c3ea8238103db2490c048814e8970735c70c42e5004e1bd08bd56507333"} Oct 31 22:19:17.304765 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.304742 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:17.308726 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.308706 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:17.401097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.400981 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-openshift-ca\") pod \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " Oct 31 22:19:17.401097 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401028 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aac996a-26d0-4c19-b7e0-23ff356ebe37-kube-api-access-m57zw\" (UniqueName: \"kubernetes.io/projected/8aac996a-26d0-4c19-b7e0-23ff356ebe37-kube-api-access-m57zw\") pod \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " Oct 31 22:19:17.401462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401130 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-kube-api-access-xskns\" (UniqueName: \"kubernetes.io/projected/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-kube-api-access-xskns\") pod \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " Oct 31 22:19:17.401462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401314 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-xks-ca\") pod \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " Oct 31 22:19:17.401462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401378 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-openshift-ca\") pod \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\" (UID: \"8aac996a-26d0-4c19-b7e0-23ff356ebe37\") " Oct 31 22:19:17.401462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401404 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-xks-ca\") pod \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\" (UID: \"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4\") " Oct 31 22:19:17.401462 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401425 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-openshift-ca" pod "8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4" (UID: "8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:17.401935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401838 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.401935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.401909 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-openshift-ca" pod "8aac996a-26d0-4c19-b7e0-23ff356ebe37" (UID: "8aac996a-26d0-4c19-b7e0-23ff356ebe37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:17.403863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.403836 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aac996a-26d0-4c19-b7e0-23ff356ebe37-kube-api-access-m57zw" pod "8aac996a-26d0-4c19-b7e0-23ff356ebe37" (UID: "8aac996a-26d0-4c19-b7e0-23ff356ebe37"). InnerVolumeSpecName "kube-api-access-m57zw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:17.403988 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.403934 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-xks-ca" pod "8aac996a-26d0-4c19-b7e0-23ff356ebe37" (UID: "8aac996a-26d0-4c19-b7e0-23ff356ebe37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:17.404068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.404002 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-xks-ca" pod "8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4" (UID: "8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:17.405721 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.405693 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-kube-api-access-xskns" pod "8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4" (UID: "8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4"). InnerVolumeSpecName "kube-api-access-xskns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:17.503113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.503062 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m57zw\" (UniqueName: \"kubernetes.io/projected/8aac996a-26d0-4c19-b7e0-23ff356ebe37-kube-api-access-m57zw\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.503113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.503113 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xskns\" (UniqueName: \"kubernetes.io/projected/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-kube-api-access-xskns\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.503113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.503125 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.503381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.503134 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aac996a-26d0-4c19-b7e0-23ff356ebe37-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.503381 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.503143 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:17.731829 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.731725 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj"] Oct 31 22:19:17.734437 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.734414 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-b36e62-x8wjj"] Oct 31 22:19:17.984275 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:17.984244 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="502f4be6-ec90-4e19-b286-ab45137faf8a" path="/var/lib/kubelet/pods/502f4be6-ec90-4e19-b286-ab45137faf8a/volumes" Oct 31 22:19:18.090201 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.090169 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" Oct 31 22:19:18.090201 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.090183 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-961790d1-vjxh5" event={"ID":"8aac996a-26d0-4c19-b7e0-23ff356ebe37","Type":"ContainerDied","Data":"79729cb475e757ddb3cdca8c20b6519b3f193824ea0418faf99807c2fbca456d"} Oct 31 22:19:18.090489 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.090236 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79729cb475e757ddb3cdca8c20b6519b3f193824ea0418faf99807c2fbca456d" Oct 31 22:19:18.091623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.091601 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" Oct 31 22:19:18.091750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.091632 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8caf3a09-hxfkd" event={"ID":"8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4","Type":"ContainerDied","Data":"25f8ad60a659811b6b73e8fe80714765ee47f3c98440c86e17e74a1f33b3b581"} Oct 31 22:19:18.091750 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.091666 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f8ad60a659811b6b73e8fe80714765ee47f3c98440c86e17e74a1f33b3b581" Oct 31 22:19:18.623546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.623505 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" containerID="cri-o://b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00" gracePeriod=45 Oct 31 22:19:18.878029 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:18.877973 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 22:19:19.020167 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020135 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-home\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-home\") pod \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " Oct 31 22:19:19.020355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020259 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-model-cache\") pod \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " Oct 31 22:19:19.020355 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020296 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tls-certs\") pod \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " Oct 31 22:19:19.020467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020353 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tmp-dir\") pod \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " Oct 31 22:19:19.020467 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020424 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-kube-api-access-k8kq6\" (UniqueName: \"kubernetes.io/projected/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-kube-api-access-k8kq6\") pod \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " Oct 31 22:19:19.020592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020494 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-dshm\") pod \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\" (UID: \"a55beff0-3d39-41d2-9cd3-98ce50abb4ae\") " Oct 31 22:19:19.020763 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020491 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-home" pod "a55beff0-3d39-41d2-9cd3-98ce50abb4ae" (UID: "a55beff0-3d39-41d2-9cd3-98ce50abb4ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.020763 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020665 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tmp-dir" pod "a55beff0-3d39-41d2-9cd3-98ce50abb4ae" (UID: "a55beff0-3d39-41d2-9cd3-98ce50abb4ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.020932 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020888 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.020932 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.020908 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.022956 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.022925 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-dshm" pod "a55beff0-3d39-41d2-9cd3-98ce50abb4ae" (UID: "a55beff0-3d39-41d2-9cd3-98ce50abb4ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.023290 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.023252 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-kube-api-access-k8kq6" pod "a55beff0-3d39-41d2-9cd3-98ce50abb4ae" (UID: "a55beff0-3d39-41d2-9cd3-98ce50abb4ae"). InnerVolumeSpecName "kube-api-access-k8kq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:19.023423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.023392 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tls-certs" pod "a55beff0-3d39-41d2-9cd3-98ce50abb4ae" (UID: "a55beff0-3d39-41d2-9cd3-98ce50abb4ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:19.024507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.024481 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-model-cache" pod "a55beff0-3d39-41d2-9cd3-98ce50abb4ae" (UID: "a55beff0-3d39-41d2-9cd3-98ce50abb4ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:19.102073 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.102031 4294 generic.go:356] "Generic (PLEG): container finished" podID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerID="b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00" exitCode=0 Oct 31 22:19:19.102267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.102073 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" event={"ID":"a55beff0-3d39-41d2-9cd3-98ce50abb4ae","Type":"ContainerDied","Data":"b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00"} Oct 31 22:19:19.102267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.102125 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" event={"ID":"a55beff0-3d39-41d2-9cd3-98ce50abb4ae","Type":"ContainerDied","Data":"1cc4199bdeda0d16743dcaf9b8f2fa6cfa33e353a9495c79f81f23120988e37c"} Oct 31 22:19:19.102267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.102143 4294 scope.go:122] "RemoveContainer" containerID="b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00" Oct 31 22:19:19.102267 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.102088 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf" Oct 31 22:19:19.114409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.114388 4294 scope.go:122] "RemoveContainer" containerID="b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00" Oct 31 22:19:19.114709 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:19.114688 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00\": container with ID starting with b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00 not found: ID does not exist" containerID="b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00" Oct 31 22:19:19.114785 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.114719 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00"} err="failed to get container status \"b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00\": rpc error: code = NotFound desc = could not find container \"b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00\": container with ID starting with b8edd5d3d48036b7bdba5ea58e9ddd739ca85089251263fb665b636a33cd1a00 not found: ID does not exist" Oct 31 22:19:19.122576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.122546 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.122576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.122578 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.122755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.122592 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kq6\" (UniqueName: \"kubernetes.io/projected/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-kube-api-access-k8kq6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.122755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.122606 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a55beff0-3d39-41d2-9cd3-98ce50abb4ae-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:19.159492 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.159408 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf"] Oct 31 22:19:19.164880 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.164851 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-cc45a7/distinct-w4-cc45a7-kserve-fddf66cbd-vtxwf"] Oct 31 22:19:19.982801 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:19.982771 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" path="/var/lib/kubelet/pods/a55beff0-3d39-41d2-9cd3-98ce50abb4ae/volumes" Oct 31 22:19:20.790279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:20.790234 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" podUID="9889bb54-d360-402e-83e8-e175c421291c" containerName="payload-pre-processing" containerID="cri-o://3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc" gracePeriod=25 Oct 31 22:19:22.681769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:22.681732 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" podUID="9889bb54-d360-402e-83e8-e175c421291c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:19:23.106218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:23.106183 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6"] Oct 31 22:19:23.118960 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:23.118926 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5c54fc-svnf6"] Oct 31 22:19:23.985291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:23.985248 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab0a982-6c4f-4d6e-9853-ee3ddfab28af" path="/var/lib/kubelet/pods/cab0a982-6c4f-4d6e-9853-ee3ddfab28af/volumes" Oct 31 22:19:25.368058 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:25.368015 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k"] Oct 31 22:19:25.382147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:25.382110 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-cc45a7-2l86k"] Oct 31 22:19:25.984260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:25.984219 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c793834-43c8-4ce0-bbe2-09bfa4c1c482" path="/var/lib/kubelet/pods/4c793834-43c8-4ce0-bbe2-09bfa4c1c482/volumes" Oct 31 22:19:27.281770 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.281729 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt"] Oct 31 22:19:27.282288 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.282007 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" containerID="cri-o://208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8" gracePeriod=30 Oct 31 22:19:27.866370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866333 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj"] Oct 31 22:19:27.866768 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866755 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4" containerName="revoke-keys" Oct 31 22:19:27.866811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866770 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4" containerName="revoke-keys" Oct 31 22:19:27.866811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866781 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" Oct 31 22:19:27.866811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866786 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" Oct 31 22:19:27.866811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866798 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8aac996a-26d0-4c19-b7e0-23ff356ebe37" containerName="revoke-keys" Oct 31 22:19:27.866811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866803 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aac996a-26d0-4c19-b7e0-23ff356ebe37" containerName="revoke-keys" Oct 31 22:19:27.866968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866873 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4" containerName="revoke-keys" Oct 31 22:19:27.866968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866880 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55beff0-3d39-41d2-9cd3-98ce50abb4ae" containerName="main" Oct 31 22:19:27.866968 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.866886 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aac996a-26d0-4c19-b7e0-23ff356ebe37" containerName="revoke-keys" Oct 31 22:19:27.871595 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.871578 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:27.888002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.887976 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:19:27.888110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.888078 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:19:27.895680 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:27.895643 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj"] Oct 31 22:19:28.035916 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.035880 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.036102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.036023 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.036172 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.036118 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2txf\" (UniqueName: \"kubernetes.io/projected/0f4c5635-2f86-4b17-8e04-44db1c249c5e-kube-api-access-s2txf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.137025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.136923 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.137025 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.136980 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2txf\" (UniqueName: \"kubernetes.io/projected/0f4c5635-2f86-4b17-8e04-44db1c249c5e-kube-api-access-s2txf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.137285 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.137053 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.137738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.137716 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.139523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.139502 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.162421 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.162396 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2txf\" (UniqueName: \"kubernetes.io/projected/0f4c5635-2f86-4b17-8e04-44db1c249c5e-kube-api-access-s2txf\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.184035 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.184006 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:28.483294 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:28.483270 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj"] Oct 31 22:19:28.484451 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:28.484423 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4c5635_2f86_4b17_8e04_44db1c249c5e.slice/crio-9294116bfb04d255397d5f91743f3c5b416c2ddc20b4edbc41bd8bcb3c57d1f3 WatchSource:0}: Error finding container 9294116bfb04d255397d5f91743f3c5b416c2ddc20b4edbc41bd8bcb3c57d1f3: Status 404 returned error can't find the container with id 9294116bfb04d255397d5f91743f3c5b416c2ddc20b4edbc41bd8bcb3c57d1f3 Oct 31 22:19:29.174626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:29.174536 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" event={"ID":"0f4c5635-2f86-4b17-8e04-44db1c249c5e","Type":"ContainerStarted","Data":"b4f306b00d166863229061ef93829fde1a64dda811c90d9009f9448cd50abf51"} Oct 31 22:19:29.174626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:29.174607 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" event={"ID":"0f4c5635-2f86-4b17-8e04-44db1c249c5e","Type":"ContainerStarted","Data":"9294116bfb04d255397d5f91743f3c5b416c2ddc20b4edbc41bd8bcb3c57d1f3"} Oct 31 22:19:29.218755 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:29.218709 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" podStartSLOduration=2.218694153 podStartE2EDuration="2.218694153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:19:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:19:29.217114912 +0000 UTC m=+3809.342222141" watchObservedRunningTime="2026-10-31 22:19:29.218694153 +0000 UTC m=+3809.343801435" Oct 31 22:19:29.317672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:29.317630 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz"] Oct 31 22:19:29.317896 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:29.317851 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" containerID="cri-o://5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38" gracePeriod=30 Oct 31 22:19:30.182970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:30.182934 4294 generic.go:356] "Generic (PLEG): container finished" podID="0f4c5635-2f86-4b17-8e04-44db1c249c5e" containerID="b4f306b00d166863229061ef93829fde1a64dda811c90d9009f9448cd50abf51" exitCode=0 Oct 31 22:19:30.183418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:30.183017 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" event={"ID":"0f4c5635-2f86-4b17-8e04-44db1c249c5e","Type":"ContainerDied","Data":"b4f306b00d166863229061ef93829fde1a64dda811c90d9009f9448cd50abf51"} Oct 31 22:19:30.963040 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:30.963003 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt"] Oct 31 22:19:30.966768 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:30.966749 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:30.986571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:30.986542 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt"] Oct 31 22:19:31.071862 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.071825 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.072066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.071945 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdt2h\" (UniqueName: \"kubernetes.io/projected/4340d468-ff5c-4da4-ae5f-6df4780c31cd-kube-api-access-sdt2h\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.072066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.071999 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.173383 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.173346 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.173592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.173423 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdt2h\" (UniqueName: \"kubernetes.io/projected/4340d468-ff5c-4da4-ae5f-6df4780c31cd-kube-api-access-sdt2h\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.173592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.173470 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.174147 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.174117 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.175771 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.175748 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.197933 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.197907 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdt2h\" (UniqueName: \"kubernetes.io/projected/4340d468-ff5c-4da4-ae5f-6df4780c31cd-kube-api-access-sdt2h\") pod \"maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.279623 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.279582 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:31.314130 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:31.314095 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:19:31.319470 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.319438 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:31.480334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.478782 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-xks-ca\") pod \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " Oct 31 22:19:31.480334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.479084 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-openshift-ca\") pod \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " Oct 31 22:19:31.480334 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.479138 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4c5635-2f86-4b17-8e04-44db1c249c5e-kube-api-access-s2txf\" (UniqueName: \"kubernetes.io/projected/0f4c5635-2f86-4b17-8e04-44db1c249c5e-kube-api-access-s2txf\") pod \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\" (UID: \"0f4c5635-2f86-4b17-8e04-44db1c249c5e\") " Oct 31 22:19:31.483330 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.482449 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4c5635-2f86-4b17-8e04-44db1c249c5e-kube-api-access-s2txf" pod "0f4c5635-2f86-4b17-8e04-44db1c249c5e" (UID: "0f4c5635-2f86-4b17-8e04-44db1c249c5e"). InnerVolumeSpecName "kube-api-access-s2txf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:31.487325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.484656 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-xks-ca" pod "0f4c5635-2f86-4b17-8e04-44db1c249c5e" (UID: "0f4c5635-2f86-4b17-8e04-44db1c249c5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:31.487325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.484956 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-openshift-ca" pod "0f4c5635-2f86-4b17-8e04-44db1c249c5e" (UID: "0f4c5635-2f86-4b17-8e04-44db1c249c5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:31.581243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.581173 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:31.581243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.581206 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2txf\" (UniqueName: \"kubernetes.io/projected/0f4c5635-2f86-4b17-8e04-44db1c249c5e-kube-api-access-s2txf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:31.581243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.581223 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4c5635-2f86-4b17-8e04-44db1c249c5e-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:31.582417 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:31.582393 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt"] Oct 31 22:19:31.584041 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:31.584009 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4340d468_ff5c_4da4_ae5f_6df4780c31cd.slice/crio-86255f5f9007b10a569e3ef9a816b6c59d92ceb68852231c1cfcd6d2c8e091e6 WatchSource:0}: Error finding container 86255f5f9007b10a569e3ef9a816b6c59d92ceb68852231c1cfcd6d2c8e091e6: Status 404 returned error can't find the container with id 86255f5f9007b10a569e3ef9a816b6c59d92ceb68852231c1cfcd6d2c8e091e6 Oct 31 22:19:32.197205 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.197114 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" event={"ID":"4340d468-ff5c-4da4-ae5f-6df4780c31cd","Type":"ContainerStarted","Data":"0b38dcf123571b1b0a55eacdbc0f30ca3676a68ef9b14c5c54c5336d9d6b63a9"} Oct 31 22:19:32.197506 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.197477 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" event={"ID":"4340d468-ff5c-4da4-ae5f-6df4780c31cd","Type":"ContainerStarted","Data":"86255f5f9007b10a569e3ef9a816b6c59d92ceb68852231c1cfcd6d2c8e091e6"} Oct 31 22:19:32.198673 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.198644 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" Oct 31 22:19:32.198978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.198670 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj" event={"ID":"0f4c5635-2f86-4b17-8e04-44db1c249c5e","Type":"ContainerDied","Data":"9294116bfb04d255397d5f91743f3c5b416c2ddc20b4edbc41bd8bcb3c57d1f3"} Oct 31 22:19:32.198978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.198712 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9294116bfb04d255397d5f91743f3c5b416c2ddc20b4edbc41bd8bcb3c57d1f3" Oct 31 22:19:32.241249 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.241201 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" podStartSLOduration=2.241188043 podStartE2EDuration="2.241188043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:19:32.239602206 +0000 UTC m=+3812.364709435" watchObservedRunningTime="2026-10-31 22:19:32.241188043 +0000 UTC m=+3812.366295315" Oct 31 22:19:32.542469 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.542441 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 22:19:32.681885 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.681845 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" podUID="9889bb54-d360-402e-83e8-e175c421291c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 22:19:32.693140 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693105 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-podinfo\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693173 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istiod-ca-cert\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693239 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-socket\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693273 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-token\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-token\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693488 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-data\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693532 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-certs\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693620 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-kube-api-access-djtjf\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-kube-api-access-djtjf\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693654 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-credential-socket\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693686 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-envoy\") pod \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\" (UID: \"8bd43c99-e098-4f97-bba2-39fc2ae8b67d\") " Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.693859 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istiod-ca-cert" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:32.694521 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.694121 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.695592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.695564 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-data" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:32.695797 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.695774 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-socket" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:32.696324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.696274 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-credential-socket" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:32.696709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.696689 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-certs" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:32.697573 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.697552 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-envoy" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:32.698569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.698538 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-kube-api-access-djtjf" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "kube-api-access-djtjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:32.698662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.698608 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-token" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:32.698831 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.698804 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-podinfo" pod "8bd43c99-e098-4f97-bba2-39fc2ae8b67d" (UID: "8bd43c99-e098-4f97-bba2-39fc2ae8b67d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:19:32.795726 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795685 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djtjf\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-kube-api-access-djtjf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.795726 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795719 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.795726 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795732 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.795981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795740 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.795981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795752 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.795981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795760 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.795981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795767 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:32.795981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:32.795775 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bd43c99-e098-4f97-bba2-39fc2ae8b67d-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:33.206357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.206261 4294 generic.go:356] "Generic (PLEG): container finished" podID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerID="208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8" exitCode=0 Oct 31 22:19:33.206357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.206326 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" Oct 31 22:19:33.206357 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.206330 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" event={"ID":"8bd43c99-e098-4f97-bba2-39fc2ae8b67d","Type":"ContainerDied","Data":"208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8"} Oct 31 22:19:33.206917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.206382 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt" event={"ID":"8bd43c99-e098-4f97-bba2-39fc2ae8b67d","Type":"ContainerDied","Data":"91152e13921120e22764b192d0498ab28cced1c8154e1e984b65c0c6973b605c"} Oct 31 22:19:33.206917 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.206397 4294 scope.go:122] "RemoveContainer" containerID="208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8" Oct 31 22:19:33.208281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.208256 4294 generic.go:356] "Generic (PLEG): container finished" podID="4340d468-ff5c-4da4-ae5f-6df4780c31cd" containerID="0b38dcf123571b1b0a55eacdbc0f30ca3676a68ef9b14c5c54c5336d9d6b63a9" exitCode=0 Oct 31 22:19:33.208409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.208378 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" event={"ID":"4340d468-ff5c-4da4-ae5f-6df4780c31cd","Type":"ContainerDied","Data":"0b38dcf123571b1b0a55eacdbc0f30ca3676a68ef9b14c5c54c5336d9d6b63a9"} Oct 31 22:19:33.219240 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.219220 4294 scope.go:122] "RemoveContainer" containerID="0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9" Oct 31 22:19:33.254825 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.254801 4294 scope.go:122] "RemoveContainer" containerID="208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8" Oct 31 22:19:33.255144 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:33.255124 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8\": container with ID starting with 208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8 not found: ID does not exist" containerID="208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8" Oct 31 22:19:33.255225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.255151 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8"} err="failed to get container status \"208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8\": rpc error: code = NotFound desc = could not find container \"208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8\": container with ID starting with 208f53549e95e944257fba701f2dda5edb2a7e22141968e3684baa319a95f0f8 not found: ID does not exist" Oct 31 22:19:33.255225 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.255168 4294 scope.go:122] "RemoveContainer" containerID="0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9" Oct 31 22:19:33.255445 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:33.255427 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9\": container with ID starting with 0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9 not found: ID does not exist" containerID="0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9" Oct 31 22:19:33.255488 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.255451 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9"} err="failed to get container status \"0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9\": rpc error: code = NotFound desc = could not find container \"0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9\": container with ID starting with 0ddc282e7515aa22b21c2b4ce0d300d207e45647caa9b8be98985d03ffb1cfb9 not found: ID does not exist" Oct 31 22:19:33.275741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.275717 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt"] Oct 31 22:19:33.288798 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.288776 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b36e62-openshift-default-5b64c855b5-g9lkt"] Oct 31 22:19:33.983877 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:33.983834 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" path="/var/lib/kubelet/pods/8bd43c99-e098-4f97-bba2-39fc2ae8b67d/volumes" Oct 31 22:19:34.412611 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.412586 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:34.514821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.514789 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-openshift-ca\") pod \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " Oct 31 22:19:34.515001 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.514929 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4340d468-ff5c-4da4-ae5f-6df4780c31cd-kube-api-access-sdt2h\" (UniqueName: \"kubernetes.io/projected/4340d468-ff5c-4da4-ae5f-6df4780c31cd-kube-api-access-sdt2h\") pod \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " Oct 31 22:19:34.515068 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.515054 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-xks-ca\") pod \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\" (UID: \"4340d468-ff5c-4da4-ae5f-6df4780c31cd\") " Oct 31 22:19:34.515255 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.515208 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-openshift-ca" pod "4340d468-ff5c-4da4-ae5f-6df4780c31cd" (UID: "4340d468-ff5c-4da4-ae5f-6df4780c31cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:34.515474 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.515430 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.517344 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.517295 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-xks-ca" pod "4340d468-ff5c-4da4-ae5f-6df4780c31cd" (UID: "4340d468-ff5c-4da4-ae5f-6df4780c31cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:34.517459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.517376 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4340d468-ff5c-4da4-ae5f-6df4780c31cd-kube-api-access-sdt2h" pod "4340d468-ff5c-4da4-ae5f-6df4780c31cd" (UID: "4340d468-ff5c-4da4-ae5f-6df4780c31cd"). InnerVolumeSpecName "kube-api-access-sdt2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:34.569384 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.569360 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:19:34.616465 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616413 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-envoy\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.616800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616520 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-certs\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.616800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616587 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-kube-api-access-gls92\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-kube-api-access-gls92\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.616800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616622 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-token\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-token\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.616800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616692 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istiod-ca-cert\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.616800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616727 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-data\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.616800 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616763 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-credential-socket\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.617277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616860 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-certs" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:34.617277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616884 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-podinfo\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.617277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.616920 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-socket\") pod \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\" (UID: \"919aad89-5a32-4eb8-bb0a-dfaf842ac2db\") " Oct 31 22:19:34.617277 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.617211 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istiod-ca-cert" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:34.617524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.617292 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4340d468-ff5c-4da4-ae5f-6df4780c31cd-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.617524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.617329 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.617524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.617342 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdt2h\" (UniqueName: \"kubernetes.io/projected/4340d468-ff5c-4da4-ae5f-6df4780c31cd-kube-api-access-sdt2h\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.617524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.617355 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.617738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.617563 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-data" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:34.617797 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.617738 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-credential-socket" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:34.618187 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.618158 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-socket" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:34.619631 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.619610 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-kube-api-access-gls92" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "kube-api-access-gls92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:34.620157 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.620129 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-envoy" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:34.620587 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.620373 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-podinfo" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:19:34.620673 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.620628 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-token" pod "919aad89-5a32-4eb8-bb0a-dfaf842ac2db" (UID: "919aad89-5a32-4eb8-bb0a-dfaf842ac2db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:34.718320 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.718274 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.718495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.718327 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.718495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.718345 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.718495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.718359 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gls92\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-kube-api-access-gls92\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.718495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.718373 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.718495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.718386 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:34.718495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:34.718398 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919aad89-5a32-4eb8-bb0a-dfaf842ac2db-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:35.223651 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.223619 4294 generic.go:356] "Generic (PLEG): container finished" podID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerID="5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38" exitCode=0 Oct 31 22:19:35.223832 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.223670 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" Oct 31 22:19:35.223832 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.223701 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" event={"ID":"919aad89-5a32-4eb8-bb0a-dfaf842ac2db","Type":"ContainerDied","Data":"5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38"} Oct 31 22:19:35.223832 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.223758 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz" event={"ID":"919aad89-5a32-4eb8-bb0a-dfaf842ac2db","Type":"ContainerDied","Data":"51c5cdf20376836e5373d5c4116bbf8c32d5a759e4cd4783d58fb43a591e6c14"} Oct 31 22:19:35.223832 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.223788 4294 scope.go:122] "RemoveContainer" containerID="5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38" Oct 31 22:19:35.225296 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.225277 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" Oct 31 22:19:35.225296 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.225283 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt" event={"ID":"4340d468-ff5c-4da4-ae5f-6df4780c31cd","Type":"ContainerDied","Data":"86255f5f9007b10a569e3ef9a816b6c59d92ceb68852231c1cfcd6d2c8e091e6"} Oct 31 22:19:35.225490 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.225345 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86255f5f9007b10a569e3ef9a816b6c59d92ceb68852231c1cfcd6d2c8e091e6" Oct 31 22:19:35.234938 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.234920 4294 scope.go:122] "RemoveContainer" containerID="5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38" Oct 31 22:19:35.235220 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:35.235204 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38\": container with ID starting with 5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38 not found: ID does not exist" containerID="5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38" Oct 31 22:19:35.235273 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.235227 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38"} err="failed to get container status \"5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38\": rpc error: code = NotFound desc = could not find container \"5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38\": container with ID starting with 5667ae5bb3c8325d914c1339612392896320c19e245c40d84db7146894b75e38 not found: ID does not exist" Oct 31 22:19:35.280572 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.280546 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz"] Oct 31 22:19:35.293569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.293545 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-kf6vz"] Oct 31 22:19:35.984776 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:35.984739 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" path="/var/lib/kubelet/pods/919aad89-5a32-4eb8-bb0a-dfaf842ac2db/volumes" Oct 31 22:19:41.093948 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.093921 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 22:19:41.197586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.197553 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9889bb54-d360-402e-83e8-e175c421291c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9889bb54-d360-402e-83e8-e175c421291c-plugins-config-volume\") pod \"9889bb54-d360-402e-83e8-e175c421291c\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " Oct 31 22:19:41.197779 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.197617 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9889bb54-d360-402e-83e8-e175c421291c-kube-api-access-ldhd5\" (UniqueName: \"kubernetes.io/projected/9889bb54-d360-402e-83e8-e175c421291c-kube-api-access-ldhd5\") pod \"9889bb54-d360-402e-83e8-e175c421291c\" (UID: \"9889bb54-d360-402e-83e8-e175c421291c\") " Oct 31 22:19:41.197916 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.197890 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9889bb54-d360-402e-83e8-e175c421291c-plugins-config-volume" pod "9889bb54-d360-402e-83e8-e175c421291c" (UID: "9889bb54-d360-402e-83e8-e175c421291c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:41.198096 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.198081 4294 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9889bb54-d360-402e-83e8-e175c421291c-plugins-config-volume\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:41.199685 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.199661 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9889bb54-d360-402e-83e8-e175c421291c-kube-api-access-ldhd5" pod "9889bb54-d360-402e-83e8-e175c421291c" (UID: "9889bb54-d360-402e-83e8-e175c421291c"). InnerVolumeSpecName "kube-api-access-ldhd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:41.265115 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.265084 4294 generic.go:356] "Generic (PLEG): container finished" podID="9889bb54-d360-402e-83e8-e175c421291c" containerID="3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc" exitCode=0 Oct 31 22:19:41.265259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.265124 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" Oct 31 22:19:41.265259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.265166 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" event={"ID":"9889bb54-d360-402e-83e8-e175c421291c","Type":"ContainerDied","Data":"3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc"} Oct 31 22:19:41.265259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.265217 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg" event={"ID":"9889bb54-d360-402e-83e8-e175c421291c","Type":"ContainerDied","Data":"f68182762785808c5aae8f48fb91ba19b82132dfe2bdca1f961d9fd32417fcd6"} Oct 31 22:19:41.265259 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.265242 4294 scope.go:122] "RemoveContainer" containerID="3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc" Oct 31 22:19:41.275823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.275805 4294 scope.go:122] "RemoveContainer" containerID="3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc" Oct 31 22:19:41.276091 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:41.276071 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc\": container with ID starting with 3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc not found: ID does not exist" containerID="3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc" Oct 31 22:19:41.276141 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.276100 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc"} err="failed to get container status \"3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc\": rpc error: code = NotFound desc = could not find container \"3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc\": container with ID starting with 3a2fbfe73fb3e34330011f53472653912a11d45491e1265581a21e6536c98dbc not found: ID does not exist" Oct 31 22:19:41.298592 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.298571 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldhd5\" (UniqueName: \"kubernetes.io/projected/9889bb54-d360-402e-83e8-e175c421291c-kube-api-access-ldhd5\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:41.318803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.318726 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg"] Oct 31 22:19:41.320865 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.320845 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b36e62-764477887-7prdg"] Oct 31 22:19:41.983725 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:41.983690 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9889bb54-d360-402e-83e8-e175c421291c" path="/var/lib/kubelet/pods/9889bb54-d360-402e-83e8-e175c421291c/volumes" Oct 31 22:19:42.421135 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:42.421050 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj"] Oct 31 22:19:42.433945 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:42.433920 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-fb32d4-h8dlj"] Oct 31 22:19:43.983222 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:43.983192 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4c5635-2f86-4b17-8e04-44db1c249c5e" path="/var/lib/kubelet/pods/0f4c5635-2f86-4b17-8e04-44db1c249c5e/volumes" Oct 31 22:19:49.512593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:49.512555 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt"] Oct 31 22:19:49.525448 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:49.525414 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-943963-tz2pt"] Oct 31 22:19:49.983941 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:49.983906 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4340d468-ff5c-4da4-ae5f-6df4780c31cd" path="/var/lib/kubelet/pods/4340d468-ff5c-4da4-ae5f-6df4780c31cd/volumes" Oct 31 22:19:50.057542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.057498 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82"] Oct 31 22:19:50.057803 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.057756 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" containerID="cri-o://597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa" gracePeriod=30 Oct 31 22:19:50.762773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.762736 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg"] Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763149 4294 cpu_manager.go:414] "containerMap: removing container" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763162 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763172 4294 cpu_manager.go:414] "containerMap: removing container" podUID="9889bb54-d360-402e-83e8-e175c421291c" containerName="payload-pre-processing" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763177 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9889bb54-d360-402e-83e8-e175c421291c" containerName="payload-pre-processing" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763196 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763201 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763208 4294 cpu_manager.go:414] "containerMap: removing container" podUID="0f4c5635-2f86-4b17-8e04-44db1c249c5e" containerName="revoke-keys" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763213 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4c5635-2f86-4b17-8e04-44db1c249c5e" containerName="revoke-keys" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763220 4294 cpu_manager.go:414] "containerMap: removing container" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" Oct 31 22:19:50.763224 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763225 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763238 4294 cpu_manager.go:414] "containerMap: removing container" podUID="4340d468-ff5c-4da4-ae5f-6df4780c31cd" containerName="revoke-keys" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763243 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4340d468-ff5c-4da4-ae5f-6df4780c31cd" containerName="revoke-keys" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763325 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="4340d468-ff5c-4da4-ae5f-6df4780c31cd" containerName="revoke-keys" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763338 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="919aad89-5a32-4eb8-bb0a-dfaf842ac2db" containerName="istio-proxy" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763346 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="9889bb54-d360-402e-83e8-e175c421291c" containerName="payload-pre-processing" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763354 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763360 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4c5635-2f86-4b17-8e04-44db1c249c5e" containerName="revoke-keys" Oct 31 22:19:50.763634 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.763367 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd43c99-e098-4f97-bba2-39fc2ae8b67d" containerName="istio-proxy" Oct 31 22:19:50.767866 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.767848 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:50.786335 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.786279 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg"] Oct 31 22:19:50.796407 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.796386 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:19:50.796508 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.796458 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:19:50.897546 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.897515 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:50.897672 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.897645 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wlm\" (UniqueName: \"kubernetes.io/projected/73d35e20-1019-4155-868f-8e4abbde0633-kube-api-access-65wlm\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:50.897716 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.897686 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:50.998478 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.998445 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:50.998593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.998544 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wlm\" (UniqueName: \"kubernetes.io/projected/73d35e20-1019-4155-868f-8e4abbde0633-kube-api-access-65wlm\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:50.998593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.998583 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:50.999243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:50.999222 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:51.000821 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:51.000799 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:51.022973 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:51.022916 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wlm\" (UniqueName: \"kubernetes.io/projected/73d35e20-1019-4155-868f-8e4abbde0633-kube-api-access-65wlm\") pod \"maas-api-revoke-keys-e2e-worker-w1-994336-4frgg\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:51.080649 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:51.080617 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:51.377510 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:51.377485 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg"] Oct 31 22:19:51.379423 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:51.379394 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d35e20_1019_4155_868f_8e4abbde0633.slice/crio-0a72e0b24bbeb8d46fdd0c612a5446e8bc27f98ec592974311d94f56df0d8f73 WatchSource:0}: Error finding container 0a72e0b24bbeb8d46fdd0c612a5446e8bc27f98ec592974311d94f56df0d8f73: Status 404 returned error can't find the container with id 0a72e0b24bbeb8d46fdd0c612a5446e8bc27f98ec592974311d94f56df0d8f73 Oct 31 22:19:52.341958 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:52.341924 4294 generic.go:356] "Generic (PLEG): container finished" podID="73d35e20-1019-4155-868f-8e4abbde0633" containerID="4b4a59520e249ad641b67e299e72583066822bea0fb9169d15ed8a672bed9788" exitCode=0 Oct 31 22:19:52.342373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:52.342016 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" event={"ID":"73d35e20-1019-4155-868f-8e4abbde0633","Type":"ContainerDied","Data":"4b4a59520e249ad641b67e299e72583066822bea0fb9169d15ed8a672bed9788"} Oct 31 22:19:52.342373 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:52.342068 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" event={"ID":"73d35e20-1019-4155-868f-8e4abbde0633","Type":"ContainerStarted","Data":"0a72e0b24bbeb8d46fdd0c612a5446e8bc27f98ec592974311d94f56df0d8f73"} Oct 31 22:19:53.029541 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.029507 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq"] Oct 31 22:19:53.032325 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.032284 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ef0d0e1-7hqjq"] Oct 31 22:19:53.476317 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.476276 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:53.624260 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.624179 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73d35e20-1019-4155-868f-8e4abbde0633-kube-api-access-65wlm\" (UniqueName: \"kubernetes.io/projected/73d35e20-1019-4155-868f-8e4abbde0633-kube-api-access-65wlm\") pod \"73d35e20-1019-4155-868f-8e4abbde0633\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " Oct 31 22:19:53.624453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.624318 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-xks-ca\") pod \"73d35e20-1019-4155-868f-8e4abbde0633\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " Oct 31 22:19:53.624453 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.624367 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-openshift-ca\") pod \"73d35e20-1019-4155-868f-8e4abbde0633\" (UID: \"73d35e20-1019-4155-868f-8e4abbde0633\") " Oct 31 22:19:53.624849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.624822 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-openshift-ca" pod "73d35e20-1019-4155-868f-8e4abbde0633" (UID: "73d35e20-1019-4155-868f-8e4abbde0633"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:53.627107 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.627072 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73d35e20-1019-4155-868f-8e4abbde0633-kube-api-access-65wlm" pod "73d35e20-1019-4155-868f-8e4abbde0633" (UID: "73d35e20-1019-4155-868f-8e4abbde0633"). InnerVolumeSpecName "kube-api-access-65wlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:53.627363 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.627330 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-xks-ca" pod "73d35e20-1019-4155-868f-8e4abbde0633" (UID: "73d35e20-1019-4155-868f-8e4abbde0633"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:19:53.725848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.725812 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:53.725848 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.725845 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65wlm\" (UniqueName: \"kubernetes.io/projected/73d35e20-1019-4155-868f-8e4abbde0633-kube-api-access-65wlm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:53.726059 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.725859 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73d35e20-1019-4155-868f-8e4abbde0633-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:53.983479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:53.983450 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf1384d-e0b7-46cc-973a-07d7f4231b4d" path="/var/lib/kubelet/pods/aaf1384d-e0b7-46cc-973a-07d7f4231b4d/volumes" Oct 31 22:19:54.356023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:54.355939 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" Oct 31 22:19:54.356023 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:54.355971 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg" event={"ID":"73d35e20-1019-4155-868f-8e4abbde0633","Type":"ContainerDied","Data":"0a72e0b24bbeb8d46fdd0c612a5446e8bc27f98ec592974311d94f56df0d8f73"} Oct 31 22:19:54.356218 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:54.356030 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a72e0b24bbeb8d46fdd0c612a5446e8bc27f98ec592974311d94f56df0d8f73" Oct 31 22:19:55.306754 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.306726 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 22:19:55.363033 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.362964 4294 generic.go:356] "Generic (PLEG): container finished" podID="5d6dff6d-171c-4611-924d-034020433dd6" containerID="597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa" exitCode=0 Oct 31 22:19:55.363033 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.363007 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" Oct 31 22:19:55.363215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.363042 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" event={"ID":"5d6dff6d-171c-4611-924d-034020433dd6","Type":"ContainerDied","Data":"597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa"} Oct 31 22:19:55.363215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.363105 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82" event={"ID":"5d6dff6d-171c-4611-924d-034020433dd6","Type":"ContainerDied","Data":"95e2ac022a059122363cd9b09f7e6ee8d4e30836d23d346b1f41bc0906c6b516"} Oct 31 22:19:55.363215 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.363128 4294 scope.go:122] "RemoveContainer" containerID="597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa" Oct 31 22:19:55.377818 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.377795 4294 scope.go:122] "RemoveContainer" containerID="bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0" Oct 31 22:19:55.387370 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.387353 4294 scope.go:122] "RemoveContainer" containerID="597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa" Oct 31 22:19:55.387632 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:55.387613 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa\": container with ID starting with 597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa not found: ID does not exist" containerID="597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa" Oct 31 22:19:55.387715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.387643 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa"} err="failed to get container status \"597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa\": rpc error: code = NotFound desc = could not find container \"597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa\": container with ID starting with 597505e8177ef8021585510af9904115df718097d985f42b5a5323d1750147fa not found: ID does not exist" Oct 31 22:19:55.387715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.387664 4294 scope.go:122] "RemoveContainer" containerID="bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0" Oct 31 22:19:55.387889 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:19:55.387872 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0\": container with ID starting with bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0 not found: ID does not exist" containerID="bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0" Oct 31 22:19:55.387949 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.387897 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0"} err="failed to get container status \"bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0\": rpc error: code = NotFound desc = could not find container \"bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0\": container with ID starting with bf4a260dd2865f93bb33bb71985283e45eae9c0db8c01ff86603a55174000dc0 not found: ID does not exist" Oct 31 22:19:55.442243 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442214 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-certs\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442253 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-socket\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442342 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-data\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442369 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6dff6d-171c-4611-924d-034020433dd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6dff6d-171c-4611-924d-034020433dd6-istiod-ca-cert\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442414 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442408 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-istio-token\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-istio-token\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442467 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-envoy\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442503 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-certs" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:55.442639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442508 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-kube-api-access-wd4hf\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-kube-api-access-wd4hf\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442635 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-socket" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:55.442811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442649 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d6dff6d-171c-4611-924d-034020433dd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6dff6d-171c-4611-924d-034020433dd6-istio-podinfo\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.442811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442658 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-data" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:55.442811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442728 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6dff6d-171c-4611-924d-034020433dd6-istiod-ca-cert" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:19:55.442811 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442754 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-credential-socket\") pod \"5d6dff6d-171c-4611-924d-034020433dd6\" (UID: \"5d6dff6d-171c-4611-924d-034020433dd6\") " Oct 31 22:19:55.443060 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.442984 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-credential-socket" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:55.443180 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.443156 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.443180 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.443183 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.443390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.443199 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.443390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.443212 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.443390 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.443225 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6dff6d-171c-4611-924d-034020433dd6-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.444709 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.444680 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d6dff6d-171c-4611-924d-034020433dd6-istio-podinfo" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:19:55.444807 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.444678 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-envoy" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:19:55.444807 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.444745 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-istio-token" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:55.444807 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.444762 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-kube-api-access-wd4hf" pod "5d6dff6d-171c-4611-924d-034020433dd6" (UID: "5d6dff6d-171c-4611-924d-034020433dd6"). InnerVolumeSpecName "kube-api-access-wd4hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:19:55.525427 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.525394 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk"] Oct 31 22:19:55.536620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.536595 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz"] Oct 31 22:19:55.544855 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.544831 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.544958 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.544859 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6dff6d-171c-4611-924d-034020433dd6-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.544958 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.544873 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4hf\" (UniqueName: \"kubernetes.io/projected/5d6dff6d-171c-4611-924d-034020433dd6-kube-api-access-wd4hf\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.544958 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.544887 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6dff6d-171c-4611-924d-034020433dd6-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:19:55.712819 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.712782 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82"] Oct 31 22:19:55.718591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.718567 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-fb32d4-openshift-default-77bcc7c457-28h82"] Oct 31 22:19:55.982700 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:55.982672 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6dff6d-171c-4611-924d-034020433dd6" path="/var/lib/kubelet/pods/5d6dff6d-171c-4611-924d-034020433dd6/volumes" Oct 31 22:19:57.685109 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.685077 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682"] Oct 31 22:19:57.685517 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.685456 4294 cpu_manager.go:414] "containerMap: removing container" podUID="73d35e20-1019-4155-868f-8e4abbde0633" containerName="revoke-keys" Oct 31 22:19:57.685517 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.685469 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73d35e20-1019-4155-868f-8e4abbde0633" containerName="revoke-keys" Oct 31 22:19:57.685517 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.685481 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" Oct 31 22:19:57.685517 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.685486 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" Oct 31 22:19:57.685646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.685557 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" Oct 31 22:19:57.685646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.685566 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="73d35e20-1019-4155-868f-8e4abbde0633" containerName="revoke-keys" Oct 31 22:19:57.690242 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.690226 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.708728 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.708700 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682"] Oct 31 22:19:57.718567 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.718541 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:19:57.718691 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.718607 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:19:57.867387 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.867355 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzgsr\" (UniqueName: \"kubernetes.io/projected/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-kube-api-access-kzgsr\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.867570 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.867419 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.867570 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.867505 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.968982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.968899 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzgsr\" (UniqueName: \"kubernetes.io/projected/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-kube-api-access-kzgsr\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.968982 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.968977 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.969203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.968999 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.969829 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.969800 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.971354 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.971328 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:57.995769 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:57.995742 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzgsr\" (UniqueName: \"kubernetes.io/projected/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-kube-api-access-kzgsr\") pod \"maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:58.003422 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:58.003402 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:19:58.301450 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:58.301422 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682"] Oct 31 22:19:58.303534 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:19:58.303503 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94825b8_770d_48f9_ab6f_bbee0c7a19c4.slice/crio-faa60bf3de3f3ecd5e597c5506d76587597a65bf6a6018d1b5f35acf6f7c3e86 WatchSource:0}: Error finding container faa60bf3de3f3ecd5e597c5506d76587597a65bf6a6018d1b5f35acf6f7c3e86: Status 404 returned error can't find the container with id faa60bf3de3f3ecd5e597c5506d76587597a65bf6a6018d1b5f35acf6f7c3e86 Oct 31 22:19:58.383801 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:58.383773 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" event={"ID":"e94825b8-770d-48f9-ab6f-bbee0c7a19c4","Type":"ContainerStarted","Data":"faa60bf3de3f3ecd5e597c5506d76587597a65bf6a6018d1b5f35acf6f7c3e86"} Oct 31 22:19:59.391619 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:59.391583 4294 generic.go:356] "Generic (PLEG): container finished" podID="e94825b8-770d-48f9-ab6f-bbee0c7a19c4" containerID="df6451cd0f6eedf58b955da06c743004452393bc8d51b7d4e16b8dbf3380bed1" exitCode=0 Oct 31 22:19:59.392007 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:19:59.391656 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" event={"ID":"e94825b8-770d-48f9-ab6f-bbee0c7a19c4","Type":"ContainerDied","Data":"df6451cd0f6eedf58b955da06c743004452393bc8d51b7d4e16b8dbf3380bed1"} Oct 31 22:20:00.530379 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.530354 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:20:00.693555 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.693479 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-xks-ca\") pod \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " Oct 31 22:20:00.693717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.693561 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-kube-api-access-kzgsr\" (UniqueName: \"kubernetes.io/projected/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-kube-api-access-kzgsr\") pod \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " Oct 31 22:20:00.693717 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.693595 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-openshift-ca\") pod \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\" (UID: \"e94825b8-770d-48f9-ab6f-bbee0c7a19c4\") " Oct 31 22:20:00.694022 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.693999 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-openshift-ca" pod "e94825b8-770d-48f9-ab6f-bbee0c7a19c4" (UID: "e94825b8-770d-48f9-ab6f-bbee0c7a19c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:00.695551 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.695528 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-xks-ca" pod "e94825b8-770d-48f9-ab6f-bbee0c7a19c4" (UID: "e94825b8-770d-48f9-ab6f-bbee0c7a19c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:00.695688 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.695664 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-kube-api-access-kzgsr" pod "e94825b8-770d-48f9-ab6f-bbee0c7a19c4" (UID: "e94825b8-770d-48f9-ab6f-bbee0c7a19c4"). InnerVolumeSpecName "kube-api-access-kzgsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:00.794950 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.794907 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzgsr\" (UniqueName: \"kubernetes.io/projected/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-kube-api-access-kzgsr\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:00.794950 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.794946 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:00.795083 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:00.794960 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e94825b8-770d-48f9-ab6f-bbee0c7a19c4-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:01.021929 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.021653 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx"] Oct 31 22:20:01.022160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.022138 4294 cpu_manager.go:414] "containerMap: removing container" podUID="e94825b8-770d-48f9-ab6f-bbee0c7a19c4" containerName="revoke-keys" Oct 31 22:20:01.022239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.022161 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e94825b8-770d-48f9-ab6f-bbee0c7a19c4" containerName="revoke-keys" Oct 31 22:20:01.022239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.022214 4294 cpu_manager.go:414] "containerMap: removing container" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" Oct 31 22:20:01.022239 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.022221 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" Oct 31 22:20:01.022646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.022295 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6dff6d-171c-4611-924d-034020433dd6" containerName="istio-proxy" Oct 31 22:20:01.022646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.022321 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="e94825b8-770d-48f9-ab6f-bbee0c7a19c4" containerName="revoke-keys" Oct 31 22:20:01.025433 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.025416 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.049333 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.049294 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx"] Oct 31 22:20:01.198321 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.198263 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.198495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.198405 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.198495 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.198441 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v696z\" (UniqueName: \"kubernetes.io/projected/2b61b5aa-913a-40be-900b-244b9706cf03-kube-api-access-v696z\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.299966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.299889 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v696z\" (UniqueName: \"kubernetes.io/projected/2b61b5aa-913a-40be-900b-244b9706cf03-kube-api-access-v696z\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.299966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.299957 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.300145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.300078 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.300598 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.300578 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.302576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.302547 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.318995 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:01.318950 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:20:01.324756 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.324731 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v696z\" (UniqueName: \"kubernetes.io/projected/2b61b5aa-913a-40be-900b-244b9706cf03-kube-api-access-v696z\") pod \"maas-api-revoke-sub-b3612f78-z6zhx\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.338587 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.338562 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:01.408959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.408928 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" event={"ID":"e94825b8-770d-48f9-ab6f-bbee0c7a19c4","Type":"ContainerDied","Data":"faa60bf3de3f3ecd5e597c5506d76587597a65bf6a6018d1b5f35acf6f7c3e86"} Oct 31 22:20:01.409086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.408979 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa60bf3de3f3ecd5e597c5506d76587597a65bf6a6018d1b5f35acf6f7c3e86" Oct 31 22:20:01.409086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.408943 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682" Oct 31 22:20:01.643500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:01.643477 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx"] Oct 31 22:20:01.645885 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:20:01.645857 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b61b5aa_913a_40be_900b_244b9706cf03.slice/crio-9f91978020aedd45a3ee558b4bd9a841dd19dcaf4137b16e3655f4818af824cb WatchSource:0}: Error finding container 9f91978020aedd45a3ee558b4bd9a841dd19dcaf4137b16e3655f4818af824cb: Status 404 returned error can't find the container with id 9f91978020aedd45a3ee558b4bd9a841dd19dcaf4137b16e3655f4818af824cb Oct 31 22:20:02.416032 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:02.415996 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" event={"ID":"2b61b5aa-913a-40be-900b-244b9706cf03","Type":"ContainerStarted","Data":"180eda0da8bfcaa669f014ac1fea71a0983bef0e5416b4bc84406bc056a61a36"} Oct 31 22:20:02.416241 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:02.416057 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" event={"ID":"2b61b5aa-913a-40be-900b-244b9706cf03","Type":"ContainerStarted","Data":"9f91978020aedd45a3ee558b4bd9a841dd19dcaf4137b16e3655f4818af824cb"} Oct 31 22:20:02.459528 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:02.459481 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" podStartSLOduration=2.459464768 podStartE2EDuration="2.459464768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:20:02.457142756 +0000 UTC m=+3842.582249987" watchObservedRunningTime="2026-10-31 22:20:02.459464768 +0000 UTC m=+3842.584571999" Oct 31 22:20:03.424143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:03.424105 4294 generic.go:356] "Generic (PLEG): container finished" podID="2b61b5aa-913a-40be-900b-244b9706cf03" containerID="180eda0da8bfcaa669f014ac1fea71a0983bef0e5416b4bc84406bc056a61a36" exitCode=0 Oct 31 22:20:03.424588 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:03.424192 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" event={"ID":"2b61b5aa-913a-40be-900b-244b9706cf03","Type":"ContainerDied","Data":"180eda0da8bfcaa669f014ac1fea71a0983bef0e5416b4bc84406bc056a61a36"} Oct 31 22:20:04.287782 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.287751 4294 scope.go:122] "RemoveContainer" containerID="24938d4db348c162d76f2c4d2ba487cde3e183e71dadf7042c3799f7f3f4bf49" Oct 31 22:20:04.297253 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.297231 4294 scope.go:122] "RemoveContainer" containerID="ffa6a730e2e2a3e74e96fc52f223519a40175521a34ee494e14bf82dbacb4a86" Oct 31 22:20:04.306254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.306238 4294 scope.go:122] "RemoveContainer" containerID="8e3a6ad107a85346d3184e4098dd943a4ca1b8448dcfa30ebf0beb008347f867" Oct 31 22:20:04.582947 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.582917 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:04.732431 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.732396 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-openshift-ca\") pod \"2b61b5aa-913a-40be-900b-244b9706cf03\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " Oct 31 22:20:04.732614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.732467 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-xks-ca\") pod \"2b61b5aa-913a-40be-900b-244b9706cf03\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " Oct 31 22:20:04.732614 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.732605 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b61b5aa-913a-40be-900b-244b9706cf03-kube-api-access-v696z\" (UniqueName: \"kubernetes.io/projected/2b61b5aa-913a-40be-900b-244b9706cf03-kube-api-access-v696z\") pod \"2b61b5aa-913a-40be-900b-244b9706cf03\" (UID: \"2b61b5aa-913a-40be-900b-244b9706cf03\") " Oct 31 22:20:04.732762 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.732740 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-openshift-ca" pod "2b61b5aa-913a-40be-900b-244b9706cf03" (UID: "2b61b5aa-913a-40be-900b-244b9706cf03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:04.733014 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.732994 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:04.734635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.734605 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-xks-ca" pod "2b61b5aa-913a-40be-900b-244b9706cf03" (UID: "2b61b5aa-913a-40be-900b-244b9706cf03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:04.734741 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.734664 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b61b5aa-913a-40be-900b-244b9706cf03-kube-api-access-v696z" pod "2b61b5aa-913a-40be-900b-244b9706cf03" (UID: "2b61b5aa-913a-40be-900b-244b9706cf03"). InnerVolumeSpecName "kube-api-access-v696z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:04.833754 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.833678 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v696z\" (UniqueName: \"kubernetes.io/projected/2b61b5aa-913a-40be-900b-244b9706cf03-kube-api-access-v696z\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:04.833754 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:04.833708 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b61b5aa-913a-40be-900b-244b9706cf03-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:05.436823 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:05.436791 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" Oct 31 22:20:05.437027 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:05.436795 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3612f78-z6zhx" event={"ID":"2b61b5aa-913a-40be-900b-244b9706cf03","Type":"ContainerDied","Data":"9f91978020aedd45a3ee558b4bd9a841dd19dcaf4137b16e3655f4818af824cb"} Oct 31 22:20:05.437027 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:05.436936 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f91978020aedd45a3ee558b4bd9a841dd19dcaf4137b16e3655f4818af824cb" Oct 31 22:20:10.545506 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.545437 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" containerID="cri-o://33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa" gracePeriod=45 Oct 31 22:20:10.555591 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.555518 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" containerID="cri-o://f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6" gracePeriod=45 Oct 31 22:20:10.651005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.650967 4294 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": dial tcp 10.134.0.44:8000: connect: connection refused" Oct 31 22:20:10.817254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.817227 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 22:20:10.838576 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.838548 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 22:20:10.896244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.896203 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a41c1a24-0bf9-42cf-aaca-ef77b311c000-kube-api-access-f5p7j\" (UniqueName: \"kubernetes.io/projected/a41c1a24-0bf9-42cf-aaca-ef77b311c000-kube-api-access-f5p7j\") pod \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " Oct 31 22:20:10.896481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.896267 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tmp-dir\") pod \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " Oct 31 22:20:10.896481 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.896432 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-model-cache\") pod \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " Oct 31 22:20:10.896626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.896508 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-home\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-home\") pod \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " Oct 31 22:20:10.896626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.896546 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-dshm\") pod \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " Oct 31 22:20:10.896626 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.896599 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tls-certs\" (UniqueName: \"kubernetes.io/secret/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tls-certs\") pod \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\" (UID: \"a41c1a24-0bf9-42cf-aaca-ef77b311c000\") " Oct 31 22:20:10.897645 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.897612 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-model-cache" pod "a41c1a24-0bf9-42cf-aaca-ef77b311c000" (UID: "a41c1a24-0bf9-42cf-aaca-ef77b311c000"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:10.897909 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.897874 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tmp-dir" pod "a41c1a24-0bf9-42cf-aaca-ef77b311c000" (UID: "a41c1a24-0bf9-42cf-aaca-ef77b311c000"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:10.898137 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.898109 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-home" pod "a41c1a24-0bf9-42cf-aaca-ef77b311c000" (UID: "a41c1a24-0bf9-42cf-aaca-ef77b311c000"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:10.900028 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.900000 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tls-certs" pod "a41c1a24-0bf9-42cf-aaca-ef77b311c000" (UID: "a41c1a24-0bf9-42cf-aaca-ef77b311c000"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:10.900479 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.900458 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41c1a24-0bf9-42cf-aaca-ef77b311c000-kube-api-access-f5p7j" pod "a41c1a24-0bf9-42cf-aaca-ef77b311c000" (UID: "a41c1a24-0bf9-42cf-aaca-ef77b311c000"). InnerVolumeSpecName "kube-api-access-f5p7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:10.901063 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.901035 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-dshm" pod "a41c1a24-0bf9-42cf-aaca-ef77b311c000" (UID: "a41c1a24-0bf9-42cf-aaca-ef77b311c000"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:10.997711 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.997675 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-dshm\") pod \"f2ee6869-0354-4185-840d-adee55cc4168\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " Oct 31 22:20:10.997894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.997768 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2ee6869-0354-4185-840d-adee55cc4168-tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ee6869-0354-4185-840d-adee55cc4168-tls-certs\") pod \"f2ee6869-0354-4185-840d-adee55cc4168\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " Oct 31 22:20:10.997894 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.997833 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-tmp-dir\") pod \"f2ee6869-0354-4185-840d-adee55cc4168\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " Oct 31 22:20:10.998029 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.997940 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ee6869-0354-4185-840d-adee55cc4168-kube-api-access-749tm\" (UniqueName: \"kubernetes.io/projected/f2ee6869-0354-4185-840d-adee55cc4168-kube-api-access-749tm\") pod \"f2ee6869-0354-4185-840d-adee55cc4168\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " Oct 31 22:20:10.998112 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998089 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-tmp-dir" pod "f2ee6869-0354-4185-840d-adee55cc4168" (UID: "f2ee6869-0354-4185-840d-adee55cc4168"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:10.998337 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998292 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-home\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-home\") pod \"f2ee6869-0354-4185-840d-adee55cc4168\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " Oct 31 22:20:10.998574 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998542 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-model-cache\") pod \"f2ee6869-0354-4185-840d-adee55cc4168\" (UID: \"f2ee6869-0354-4185-840d-adee55cc4168\") " Oct 31 22:20:10.998660 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998539 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-home" pod "f2ee6869-0354-4185-840d-adee55cc4168" (UID: "f2ee6869-0354-4185-840d-adee55cc4168"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998770 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-model-cache" pod "f2ee6869-0354-4185-840d-adee55cc4168" (UID: "f2ee6869-0354-4185-840d-adee55cc4168"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998945 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5p7j\" (UniqueName: \"kubernetes.io/projected/a41c1a24-0bf9-42cf-aaca-ef77b311c000-kube-api-access-f5p7j\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998958 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998970 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998980 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.998993 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.999003 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.999011 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.999018 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a41c1a24-0bf9-42cf-aaca-ef77b311c000-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:10.999042 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:10.999026 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a41c1a24-0bf9-42cf-aaca-ef77b311c000-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.000459 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.000432 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-dshm" pod "f2ee6869-0354-4185-840d-adee55cc4168" (UID: "f2ee6869-0354-4185-840d-adee55cc4168"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:11.000542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.000470 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ee6869-0354-4185-840d-adee55cc4168-kube-api-access-749tm" pod "f2ee6869-0354-4185-840d-adee55cc4168" (UID: "f2ee6869-0354-4185-840d-adee55cc4168"). InnerVolumeSpecName "kube-api-access-749tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:11.000542 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.000518 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2ee6869-0354-4185-840d-adee55cc4168-tls-certs" pod "f2ee6869-0354-4185-840d-adee55cc4168" (UID: "f2ee6869-0354-4185-840d-adee55cc4168"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:11.099833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.099750 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2ee6869-0354-4185-840d-adee55cc4168-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.099833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.099789 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2ee6869-0354-4185-840d-adee55cc4168-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.099833 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.099807 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-749tm\" (UniqueName: \"kubernetes.io/projected/f2ee6869-0354-4185-840d-adee55cc4168-kube-api-access-749tm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:11.473043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.472964 4294 generic.go:356] "Generic (PLEG): container finished" podID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerID="33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa" exitCode=0 Oct 31 22:20:11.473043 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.473010 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" Oct 31 22:20:11.473287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.473048 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" event={"ID":"a41c1a24-0bf9-42cf-aaca-ef77b311c000","Type":"ContainerDied","Data":"33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa"} Oct 31 22:20:11.473287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.473101 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk" event={"ID":"a41c1a24-0bf9-42cf-aaca-ef77b311c000","Type":"ContainerDied","Data":"abfee9ee58058561cdeae4413e9c42be43d4d6d58910593a1d0914ceadbd2b3d"} Oct 31 22:20:11.473287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.473124 4294 scope.go:122] "RemoveContainer" containerID="33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa" Oct 31 22:20:11.474450 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.474420 4294 generic.go:356] "Generic (PLEG): container finished" podID="f2ee6869-0354-4185-840d-adee55cc4168" containerID="f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6" exitCode=0 Oct 31 22:20:11.474579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.474460 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" Oct 31 22:20:11.474579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.474498 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" event={"ID":"f2ee6869-0354-4185-840d-adee55cc4168","Type":"ContainerDied","Data":"f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6"} Oct 31 22:20:11.474579 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.474537 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz" event={"ID":"f2ee6869-0354-4185-840d-adee55cc4168","Type":"ContainerDied","Data":"e720a2596985d10b92a789cdf91cd96d220654766fc251bdab9769bf0ac70b23"} Oct 31 22:20:11.485639 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.485621 4294 scope.go:122] "RemoveContainer" containerID="33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa" Oct 31 22:20:11.485922 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:11.485898 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa\": container with ID starting with 33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa not found: ID does not exist" containerID="33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa" Oct 31 22:20:11.485984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.485933 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa"} err="failed to get container status \"33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa\": rpc error: code = NotFound desc = could not find container \"33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa\": container with ID starting with 33577a894a72cdfb3acc83323515f4833c69323a153128c0a0267e31b9f6f8fa not found: ID does not exist" Oct 31 22:20:11.485984 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.485958 4294 scope.go:122] "RemoveContainer" containerID="f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6" Oct 31 22:20:11.495265 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.495246 4294 scope.go:122] "RemoveContainer" containerID="f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6" Oct 31 22:20:11.495541 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:11.495522 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6\": container with ID starting with f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6 not found: ID does not exist" containerID="f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6" Oct 31 22:20:11.495657 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.495546 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6"} err="failed to get container status \"f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6\": rpc error: code = NotFound desc = could not find container \"f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6\": container with ID starting with f234336dfdf64a48fad2c495a9f6d5b892a252638cf5a96551816fad861c70b6 not found: ID does not exist" Oct 31 22:20:11.527054 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.527025 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk"] Oct 31 22:20:11.541360 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.541332 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-994336/premium-w1-994336-kserve-6bfc5d4c67-pvflk"] Oct 31 22:20:11.568966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.568938 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz"] Oct 31 22:20:11.579608 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.579578 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-994336/simulator-w1-994336-kserve-d4cc9dff4-szsbz"] Oct 31 22:20:11.619593 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.619570 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682"] Oct 31 22:20:11.621624 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.621600 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-56a6d4-kb682"] Oct 31 22:20:11.984099 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.984057 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" path="/var/lib/kubelet/pods/a41c1a24-0bf9-42cf-aaca-ef77b311c000/volumes" Oct 31 22:20:11.984627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.984606 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e94825b8-770d-48f9-ab6f-bbee0c7a19c4" path="/var/lib/kubelet/pods/e94825b8-770d-48f9-ab6f-bbee0c7a19c4/volumes" Oct 31 22:20:11.984991 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:11.984977 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ee6869-0354-4185-840d-adee55cc4168" path="/var/lib/kubelet/pods/f2ee6869-0354-4185-840d-adee55cc4168/volumes" Oct 31 22:20:15.497004 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:15.496960 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg"] Oct 31 22:20:15.509607 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:15.509582 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-994336-4frgg"] Oct 31 22:20:15.983005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:15.982968 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73d35e20-1019-4155-868f-8e4abbde0633" path="/var/lib/kubelet/pods/73d35e20-1019-4155-868f-8e4abbde0633/volumes" Oct 31 22:20:19.828194 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:19.828158 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz"] Oct 31 22:20:19.828748 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:19.828427 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" containerID="cri-o://d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331" gracePeriod=30 Oct 31 22:20:22.634637 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:22.634604 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6"] Oct 31 22:20:22.635192 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:22.634871 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" containerID="cri-o://2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297" gracePeriod=30 Oct 31 22:20:25.074248 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.074225 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 22:20:25.144732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.144668 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-kube-api-access-kd7w6\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-kube-api-access-kd7w6\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.144732 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.144725 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-socket\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.144898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.144753 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-podinfo\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.144898 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.144857 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-credential-socket\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.144981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.144897 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istiod-ca-cert\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.144981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.144933 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-envoy\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.144981 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.144962 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-certs\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.145145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145019 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-data\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.145145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145061 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-token\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-token\") pod \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\" (UID: \"fc39eac6-d68f-4420-87d7-48f4c0fbbe96\") " Oct 31 22:20:25.145145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145081 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-socket" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:25.145145 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145090 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-credential-socket" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:25.145423 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145274 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-certs" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:25.145485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145430 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-data" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:25.145485 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145462 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istiod-ca-cert" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:25.145571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145518 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.145571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145530 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.145571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145539 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.145571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.145547 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.146962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.146935 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-podinfo" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:20:25.147174 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.147158 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-kube-api-access-kd7w6" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "kube-api-access-kd7w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:25.147291 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.147278 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-token" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:25.147583 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.147560 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-envoy" pod "fc39eac6-d68f-4420-87d7-48f4c0fbbe96" (UID: "fc39eac6-d68f-4420-87d7-48f4c0fbbe96"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:25.246200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.246170 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.246200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.246196 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd7w6\" (UniqueName: \"kubernetes.io/projected/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-kube-api-access-kd7w6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.246200 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.246206 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.246435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.246214 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.246435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.246223 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc39eac6-d68f-4420-87d7-48f4c0fbbe96-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:25.289418 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.289389 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md"] Oct 31 22:20:25.306000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.305973 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf"] Oct 31 22:20:25.561635 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.561605 4294 generic.go:356] "Generic (PLEG): container finished" podID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerID="d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331" exitCode=0 Oct 31 22:20:25.561804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.561653 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" Oct 31 22:20:25.561804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.561688 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" event={"ID":"fc39eac6-d68f-4420-87d7-48f4c0fbbe96","Type":"ContainerDied","Data":"d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331"} Oct 31 22:20:25.561804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.561742 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz" event={"ID":"fc39eac6-d68f-4420-87d7-48f4c0fbbe96","Type":"ContainerDied","Data":"9d5366f7429a6ed7ec5d8f1662d404057a785713fe4b40f6e9cb5fe23ab0f096"} Oct 31 22:20:25.561804 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.561766 4294 scope.go:122] "RemoveContainer" containerID="d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331" Oct 31 22:20:25.572685 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.572669 4294 scope.go:122] "RemoveContainer" containerID="158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e" Oct 31 22:20:25.581532 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.581514 4294 scope.go:122] "RemoveContainer" containerID="d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331" Oct 31 22:20:25.581785 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:25.581767 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331\": container with ID starting with d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331 not found: ID does not exist" containerID="d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331" Oct 31 22:20:25.581830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.581793 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331"} err="failed to get container status \"d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331\": rpc error: code = NotFound desc = could not find container \"d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331\": container with ID starting with d1a76ca0a018ff13270da21a72787979caca6ac7735296b8859f136e402df331 not found: ID does not exist" Oct 31 22:20:25.581830 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.581809 4294 scope.go:122] "RemoveContainer" containerID="158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e" Oct 31 22:20:25.582049 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:25.582029 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e\": container with ID starting with 158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e not found: ID does not exist" containerID="158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e" Oct 31 22:20:25.582091 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.582055 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e"} err="failed to get container status \"158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e\": rpc error: code = NotFound desc = could not find container \"158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e\": container with ID starting with 158513aaac7aafc6b4913673ce2d33767e3fb2970f1d05b88337aaae4cddbe0e not found: ID does not exist" Oct 31 22:20:25.622213 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.622188 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz"] Oct 31 22:20:25.624525 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.624503 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-56a6d4-openshift-default-75f8d9db69-p7jkz"] Oct 31 22:20:25.983405 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:25.983377 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" path="/var/lib/kubelet/pods/fc39eac6-d68f-4420-87d7-48f4c0fbbe96/volumes" Oct 31 22:20:27.886006 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.885983 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 22:20:27.970586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970514 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-data\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.970586 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970563 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-certs\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.970795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970596 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-podinfo\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.970795 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970643 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-kube-api-access-nzqrn\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-kube-api-access-nzqrn\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.970897 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970829 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-data" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:27.970897 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970845 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-envoy\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.971000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970937 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-certs" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:27.971000 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970960 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-socket\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.971102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.970997 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea4b0b62-3214-4588-84f5-0e9717bf797c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4b0b62-3214-4588-84f5-0e9717bf797c-istiod-ca-cert\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.971102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971029 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-credential-socket\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.971102 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971064 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-token\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-token\") pod \"ea4b0b62-3214-4588-84f5-0e9717bf797c\" (UID: \"ea4b0b62-3214-4588-84f5-0e9717bf797c\") " Oct 31 22:20:27.971245 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971193 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-socket" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:27.971432 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971411 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea4b0b62-3214-4588-84f5-0e9717bf797c-istiod-ca-cert" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:27.971524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971491 4294 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:27.971524 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971512 4294 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea4b0b62-3214-4588-84f5-0e9717bf797c-istiod-ca-cert\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:27.971662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971526 4294 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-data\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:27.971662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971589 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-credential-socket" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:27.971662 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.971617 4294 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-workload-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:27.973164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.973130 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-envoy" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:27.973863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.973832 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-token" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:27.973863 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.973851 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-kube-api-access-nzqrn" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "kube-api-access-nzqrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:27.974005 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:27.973860 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-podinfo" pod "ea4b0b62-3214-4588-84f5-0e9717bf797c" (UID: "ea4b0b62-3214-4588-84f5-0e9717bf797c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 22:20:28.072276 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.072249 4294 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-credential-socket\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:28.072276 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.072276 4294 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-token\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:28.072430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.072289 4294 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-podinfo\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:28.072430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.072326 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqrn\" (UniqueName: \"kubernetes.io/projected/ea4b0b62-3214-4588-84f5-0e9717bf797c-kube-api-access-nzqrn\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:28.072430 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.072340 4294 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea4b0b62-3214-4588-84f5-0e9717bf797c-istio-envoy\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:28.580873 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.580838 4294 generic.go:356] "Generic (PLEG): container finished" podID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerID="2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297" exitCode=0 Oct 31 22:20:28.581053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.580877 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" Oct 31 22:20:28.581053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.580922 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" event={"ID":"ea4b0b62-3214-4588-84f5-0e9717bf797c","Type":"ContainerDied","Data":"2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297"} Oct 31 22:20:28.581053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.580978 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6" event={"ID":"ea4b0b62-3214-4588-84f5-0e9717bf797c","Type":"ContainerDied","Data":"b6bab8e07e687c79eb274e5e3a011f9cb7989893b8134d64149cb0dfe1631f96"} Oct 31 22:20:28.581053 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.581001 4294 scope.go:122] "RemoveContainer" containerID="2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297" Oct 31 22:20:28.590646 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.590629 4294 scope.go:122] "RemoveContainer" containerID="2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297" Oct 31 22:20:28.590914 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:28.590891 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297\": container with ID starting with 2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297 not found: ID does not exist" containerID="2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297" Oct 31 22:20:28.590991 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.590927 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297"} err="failed to get container status \"2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297\": rpc error: code = NotFound desc = could not find container \"2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297\": container with ID starting with 2d350c4e2ab5d03055111d1d09d38ab040f9f334312b511b047c11cff960c297 not found: ID does not exist" Oct 31 22:20:28.649992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.649966 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6"] Oct 31 22:20:28.662989 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:28.662964 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-994336-openshift-default-5dd7866d79-vb9h6"] Oct 31 22:20:29.984333 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:29.984289 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" path="/var/lib/kubelet/pods/ea4b0b62-3214-4588-84f5-0e9717bf797c/volumes" Oct 31 22:20:31.309614 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:31.309579 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:20:33.431627 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.431590 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl"] Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.431939 4294 cpu_manager.go:414] "containerMap: removing container" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.431951 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.431964 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.431972 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.431997 4294 cpu_manager.go:414] "containerMap: removing container" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432002 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432019 4294 cpu_manager.go:414] "containerMap: removing container" podUID="2b61b5aa-913a-40be-900b-244b9706cf03" containerName="revoke-keys" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432024 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b61b5aa-913a-40be-900b-244b9706cf03" containerName="revoke-keys" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432032 4294 cpu_manager.go:414] "containerMap: removing container" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432037 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432092 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="a41c1a24-0bf9-42cf-aaca-ef77b311c000" containerName="main" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432103 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432109 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b61b5aa-913a-40be-900b-244b9706cf03" containerName="revoke-keys" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432115 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4b0b62-3214-4588-84f5-0e9717bf797c" containerName="istio-proxy" Oct 31 22:20:33.432134 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.432123 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ee6869-0354-4185-840d-adee55cc4168" containerName="main" Oct 31 22:20:33.436786 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.436763 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.455548 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.455521 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl"] Oct 31 22:20:33.517936 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.517904 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.518126 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.517941 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.518126 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.517985 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh5rl\" (UniqueName: \"kubernetes.io/projected/7064cc92-7aea-4977-8bb8-a357e08b00ad-kube-api-access-zh5rl\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.536515 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.536487 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 22:20:33.536654 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.536625 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9hwvp\"" Oct 31 22:20:33.619066 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.619033 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.619250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.619077 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.619250 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.619157 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh5rl\" (UniqueName: \"kubernetes.io/projected/7064cc92-7aea-4977-8bb8-a357e08b00ad-kube-api-access-zh5rl\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.619802 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.619776 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.621480 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.621461 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.643696 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.643669 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh5rl\" (UniqueName: \"kubernetes.io/projected/7064cc92-7aea-4977-8bb8-a357e08b00ad-kube-api-access-zh5rl\") pod \"maas-api-revoke-sub-532c138d-bknpl\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:33.753604 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:33.753571 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:34.062315 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:34.062274 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl"] Oct 31 22:20:34.064405 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:20:34.064377 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7064cc92_7aea_4977_8bb8_a357e08b00ad.slice/crio-b8844bd59b7fcb47ae9e1b8bf2b8f99d425f400b43c50296b3188949e7d10a6f WatchSource:0}: Error finding container b8844bd59b7fcb47ae9e1b8bf2b8f99d425f400b43c50296b3188949e7d10a6f: Status 404 returned error can't find the container with id b8844bd59b7fcb47ae9e1b8bf2b8f99d425f400b43c50296b3188949e7d10a6f Oct 31 22:20:34.619516 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:34.619472 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" event={"ID":"7064cc92-7aea-4977-8bb8-a357e08b00ad","Type":"ContainerStarted","Data":"89fded2357e7e4e24b99ea52ef04d424abbbbeae33774405ce78489a2e9cfe26"} Oct 31 22:20:34.619966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:34.619552 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" event={"ID":"7064cc92-7aea-4977-8bb8-a357e08b00ad","Type":"ContainerStarted","Data":"b8844bd59b7fcb47ae9e1b8bf2b8f99d425f400b43c50296b3188949e7d10a6f"} Oct 31 22:20:34.662345 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:34.662245 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" podStartSLOduration=1.662231753 podStartE2EDuration="1.662231753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:20:34.661150762 +0000 UTC m=+3874.786257992" watchObservedRunningTime="2026-10-31 22:20:34.662231753 +0000 UTC m=+3874.787339026" Oct 31 22:20:35.626908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:35.626873 4294 generic.go:356] "Generic (PLEG): container finished" podID="7064cc92-7aea-4977-8bb8-a357e08b00ad" containerID="89fded2357e7e4e24b99ea52ef04d424abbbbeae33774405ce78489a2e9cfe26" exitCode=0 Oct 31 22:20:35.627333 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:35.626964 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" event={"ID":"7064cc92-7aea-4977-8bb8-a357e08b00ad","Type":"ContainerDied","Data":"89fded2357e7e4e24b99ea52ef04d424abbbbeae33774405ce78489a2e9cfe26"} Oct 31 22:20:36.777082 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.777057 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:36.848978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.848941 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-openshift-ca\") pod \"7064cc92-7aea-4977-8bb8-a357e08b00ad\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " Oct 31 22:20:36.848978 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.848990 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-xks-ca\") pod \"7064cc92-7aea-4977-8bb8-a357e08b00ad\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " Oct 31 22:20:36.849254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.849123 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7064cc92-7aea-4977-8bb8-a357e08b00ad-kube-api-access-zh5rl\" (UniqueName: \"kubernetes.io/projected/7064cc92-7aea-4977-8bb8-a357e08b00ad-kube-api-access-zh5rl\") pod \"7064cc92-7aea-4977-8bb8-a357e08b00ad\" (UID: \"7064cc92-7aea-4977-8bb8-a357e08b00ad\") " Oct 31 22:20:36.849435 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.849405 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-openshift-ca" pod "7064cc92-7aea-4977-8bb8-a357e08b00ad" (UID: "7064cc92-7aea-4977-8bb8-a357e08b00ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 22:20:36.851258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.851226 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-xks-ca" pod "7064cc92-7aea-4977-8bb8-a357e08b00ad" (UID: "7064cc92-7aea-4977-8bb8-a357e08b00ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:36.851409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.851328 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7064cc92-7aea-4977-8bb8-a357e08b00ad-kube-api-access-zh5rl" pod "7064cc92-7aea-4977-8bb8-a357e08b00ad" (UID: "7064cc92-7aea-4977-8bb8-a357e08b00ad"). InnerVolumeSpecName "kube-api-access-zh5rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:36.950314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.950223 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh5rl\" (UniqueName: \"kubernetes.io/projected/7064cc92-7aea-4977-8bb8-a357e08b00ad-kube-api-access-zh5rl\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:36.950314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.950256 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-openshift-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:36.950314 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:36.950269 4294 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7064cc92-7aea-4977-8bb8-a357e08b00ad-maas-api-xks-ca\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:37.640563 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:37.640530 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" Oct 31 22:20:37.640718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:37.640560 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-532c138d-bknpl" event={"ID":"7064cc92-7aea-4977-8bb8-a357e08b00ad","Type":"ContainerDied","Data":"b8844bd59b7fcb47ae9e1b8bf2b8f99d425f400b43c50296b3188949e7d10a6f"} Oct 31 22:20:37.640718 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:37.640612 4294 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8844bd59b7fcb47ae9e1b8bf2b8f99d425f400b43c50296b3188949e7d10a6f" Oct 31 22:20:40.308714 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.308671 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" containerID="cri-o://fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a" gracePeriod=45 Oct 31 22:20:40.324204 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.324166 4294 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" containerID="cri-o://f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511" gracePeriod=45 Oct 31 22:20:40.584587 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.584562 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 22:20:40.587409 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.587390 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 22:20:40.659796 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.659769 4294 generic.go:356] "Generic (PLEG): container finished" podID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerID="fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a" exitCode=0 Oct 31 22:20:40.660003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.659822 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" Oct 31 22:20:40.660003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.659858 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" event={"ID":"c5d13372-0bc0-4ffa-886b-d91540606cbe","Type":"ContainerDied","Data":"fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a"} Oct 31 22:20:40.660003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.659903 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md" event={"ID":"c5d13372-0bc0-4ffa-886b-d91540606cbe","Type":"ContainerDied","Data":"3fcfbcdf21ab8a6bd840ea28666bc7c89ff8b2db620b85a9db33ab82b1ae3cec"} Oct 31 22:20:40.660003 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.659927 4294 scope.go:122] "RemoveContainer" containerID="fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a" Oct 31 22:20:40.661448 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.661427 4294 generic.go:356] "Generic (PLEG): container finished" podID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerID="f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511" exitCode=0 Oct 31 22:20:40.661571 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.661557 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" event={"ID":"82abe779-421c-476a-8f64-bdecf9dccf3f","Type":"ContainerDied","Data":"f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511"} Oct 31 22:20:40.661630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.661582 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" event={"ID":"82abe779-421c-476a-8f64-bdecf9dccf3f","Type":"ContainerDied","Data":"23f1409705091bd3964665388cf6a0be96aca33b2b570a8d79d669389099e49f"} Oct 31 22:20:40.661630 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.661627 4294 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf" Oct 31 22:20:40.670550 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.670533 4294 scope.go:122] "RemoveContainer" containerID="fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a" Oct 31 22:20:40.670838 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:40.670815 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a\": container with ID starting with fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a not found: ID does not exist" containerID="fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a" Oct 31 22:20:40.670906 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.670843 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a"} err="failed to get container status \"fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a\": rpc error: code = NotFound desc = could not find container \"fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a\": container with ID starting with fb85a5968095c77ce30a37bc38a596cccc627482cfffe19a9a13a9781b0f7c4a not found: ID does not exist" Oct 31 22:20:40.670906 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.670860 4294 scope.go:122] "RemoveContainer" containerID="f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.685817 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82abe779-421c-476a-8f64-bdecf9dccf3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/82abe779-421c-476a-8f64-bdecf9dccf3f-tls-certs\") pod \"82abe779-421c-476a-8f64-bdecf9dccf3f\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.685906 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-dshm\") pod \"82abe779-421c-476a-8f64-bdecf9dccf3f\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.685963 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-model-cache\") pod \"c5d13372-0bc0-4ffa-886b-d91540606cbe\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686014 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-tmp-dir\") pod \"c5d13372-0bc0-4ffa-886b-d91540606cbe\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686062 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5d13372-0bc0-4ffa-886b-d91540606cbe-kube-api-access-g4c54\" (UniqueName: \"kubernetes.io/projected/c5d13372-0bc0-4ffa-886b-d91540606cbe-kube-api-access-g4c54\") pod \"c5d13372-0bc0-4ffa-886b-d91540606cbe\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686110 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-model-cache\") pod \"82abe779-421c-476a-8f64-bdecf9dccf3f\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686143 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82abe779-421c-476a-8f64-bdecf9dccf3f-kube-api-access-k5qc6\" (UniqueName: \"kubernetes.io/projected/82abe779-421c-476a-8f64-bdecf9dccf3f-kube-api-access-k5qc6\") pod \"82abe779-421c-476a-8f64-bdecf9dccf3f\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686186 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-tmp-dir\") pod \"82abe779-421c-476a-8f64-bdecf9dccf3f\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686276 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5d13372-0bc0-4ffa-886b-d91540606cbe-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d13372-0bc0-4ffa-886b-d91540606cbe-tls-certs\") pod \"c5d13372-0bc0-4ffa-886b-d91540606cbe\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686346 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-dshm\") pod \"c5d13372-0bc0-4ffa-886b-d91540606cbe\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686378 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-home\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-home\") pod \"82abe779-421c-476a-8f64-bdecf9dccf3f\" (UID: \"82abe779-421c-476a-8f64-bdecf9dccf3f\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686415 4294 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-home\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-home\") pod \"c5d13372-0bc0-4ffa-886b-d91540606cbe\" (UID: \"c5d13372-0bc0-4ffa-886b-d91540606cbe\") " Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686625 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-tmp-dir" pod "82abe779-421c-476a-8f64-bdecf9dccf3f" (UID: "82abe779-421c-476a-8f64-bdecf9dccf3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686982 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-home" pod "c5d13372-0bc0-4ffa-886b-d91540606cbe" (UID: "c5d13372-0bc0-4ffa-886b-d91540606cbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.686988 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-home" pod "82abe779-421c-476a-8f64-bdecf9dccf3f" (UID: "82abe779-421c-476a-8f64-bdecf9dccf3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.687214 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-model-cache" pod "c5d13372-0bc0-4ffa-886b-d91540606cbe" (UID: "c5d13372-0bc0-4ffa-886b-d91540606cbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.687421 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-model-cache" pod "82abe779-421c-476a-8f64-bdecf9dccf3f" (UID: "82abe779-421c-476a-8f64-bdecf9dccf3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.687432 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-tmp-dir" pod "c5d13372-0bc0-4ffa-886b-d91540606cbe" (UID: "c5d13372-0bc0-4ffa-886b-d91540606cbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.688951 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-dshm" pod "c5d13372-0bc0-4ffa-886b-d91540606cbe" (UID: "c5d13372-0bc0-4ffa-886b-d91540606cbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.691254 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82abe779-421c-476a-8f64-bdecf9dccf3f-tls-certs" pod "82abe779-421c-476a-8f64-bdecf9dccf3f" (UID: "82abe779-421c-476a-8f64-bdecf9dccf3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.691382 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-dshm" pod "82abe779-421c-476a-8f64-bdecf9dccf3f" (UID: "82abe779-421c-476a-8f64-bdecf9dccf3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.691471 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82abe779-421c-476a-8f64-bdecf9dccf3f-kube-api-access-k5qc6" pod "82abe779-421c-476a-8f64-bdecf9dccf3f" (UID: "82abe779-421c-476a-8f64-bdecf9dccf3f"). InnerVolumeSpecName "kube-api-access-k5qc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.691575 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5d13372-0bc0-4ffa-886b-d91540606cbe-tls-certs" pod "c5d13372-0bc0-4ffa-886b-d91540606cbe" (UID: "c5d13372-0bc0-4ffa-886b-d91540606cbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.692570 4294 scope.go:122] "RemoveContainer" containerID="f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.692781 4294 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5d13372-0bc0-4ffa-886b-d91540606cbe-kube-api-access-g4c54" pod "c5d13372-0bc0-4ffa-886b-d91540606cbe" (UID: "c5d13372-0bc0-4ffa-886b-d91540606cbe"). InnerVolumeSpecName "kube-api-access-g4c54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:20:40.692990 4294 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511\": container with ID starting with f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511 not found: ID does not exist" containerID="f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511" Oct 31 22:20:40.694329 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.693021 4294 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511"} err="failed to get container status \"f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511\": rpc error: code = NotFound desc = could not find container \"f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511\": container with ID starting with f78fbf8d5d92837872c90408d99bdf247cbef650fe29d639bdd83e711db06511 not found: ID does not exist" Oct 31 22:20:40.787992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.787959 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82abe779-421c-476a-8f64-bdecf9dccf3f-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.787992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.787988 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.787992 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.787997 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788004 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788013 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4c54\" (UniqueName: \"kubernetes.io/projected/c5d13372-0bc0-4ffa-886b-d91540606cbe-kube-api-access-g4c54\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788022 4294 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-model-cache\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788033 4294 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5qc6\" (UniqueName: \"kubernetes.io/projected/82abe779-421c-476a-8f64-bdecf9dccf3f-kube-api-access-k5qc6\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788043 4294 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-tmp-dir\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788055 4294 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5d13372-0bc0-4ffa-886b-d91540606cbe-tls-certs\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788065 4294 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-dshm\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788072 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82abe779-421c-476a-8f64-bdecf9dccf3f-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:40.788226 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:40.788079 4294 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5d13372-0bc0-4ffa-886b-d91540606cbe-home\") on node \"ip-10-0-136-156.ec2.internal\" DevicePath \"\"" Oct 31 22:20:41.010113 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:41.010086 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md"] Oct 31 22:20:41.021736 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:41.021711 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-106555/distinct-w3-106555-kserve-84f584fddd-758md"] Oct 31 22:20:41.048489 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:41.048465 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf"] Oct 31 22:20:41.052849 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:41.052828 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-106555/simulator-w3-106555-kserve-5975f447b-9kpgf"] Oct 31 22:20:41.983913 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:41.983875 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" path="/var/lib/kubelet/pods/82abe779-421c-476a-8f64-bdecf9dccf3f/volumes" Oct 31 22:20:41.984472 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:41.984451 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" path="/var/lib/kubelet/pods/c5d13372-0bc0-4ffa-886b-d91540606cbe/volumes" Oct 31 22:20:57.505967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:57.505932 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c695f6bc5-nbdsf_dbeb690d-72d7-44bc-bc54-e4512acd0d09/authorino/0.log" Oct 31 22:20:58.006077 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:58.006041 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-774gh_9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777/manager/0.log" Oct 31 22:20:58.179764 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:58.179738 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vpn5l_90d377a0-ae28-4f98-9020-54e37535c7e4/kuadrant-console-plugin/0.log" Oct 31 22:20:58.960292 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:20:58.960261 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ltl6h_80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5/manager/0.log" Oct 31 22:21:00.371271 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:00.371237 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7m5zl_26251b9d-2632-449f-9e30-eecdf1204060/manager/0.log" Oct 31 22:21:00.681738 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:00.681665 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:21:00.724983 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:00.724958 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zd6fn_cfc3bafa-8448-4f87-8480-f7311a31d1b4/ovn-acl-logging/0.log" Oct 31 22:21:01.309318 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:21:01.309266 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:21:02.394864 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:02.394826 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c695f6bc5-nbdsf_dbeb690d-72d7-44bc-bc54-e4512acd0d09/authorino/0.log" Oct 31 22:21:04.370185 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:04.370148 4294 scope.go:122] "RemoveContainer" containerID="725976b53ee7e10345c21db4d5b82835184c029b52158d4f1b71f58a55d74f6c" Oct 31 22:21:04.378773 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:04.378748 4294 scope.go:122] "RemoveContainer" containerID="7f61500ccd811c1af5b7a7e091ffa658e6aac1b5cf8187ef6e3c1172ddca93e9" Oct 31 22:21:13.402523 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:13.402437 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7m5zl_26251b9d-2632-449f-9e30-eecdf1204060/manager/0.log" Oct 31 22:21:15.725753 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:15.725725 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-05291d46-k4pdb_7ec6fd83-e9bd-47e7-8779-e41f41d736e8/revoke-keys/0.log" Oct 31 22:21:16.111970 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:16.111940 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2869d921-n9lhl_1158c05b-ae26-4736-b2fd-a632bb7fdf65/revoke-keys/0.log" Oct 31 22:21:16.306892 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:16.306864 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-38ac7ac6-cn68k_c8d09e7d-a810-4971-a875-048398ed219c/revoke-keys/0.log" Oct 31 22:21:16.495138 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:16.495116 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-532c138d-bknpl_7064cc92-7aea-4977-8bb8-a357e08b00ad/revoke-keys/0.log" Oct 31 22:21:16.690011 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:16.689977 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-671d8dec-hcqtk_ce717cad-91fe-441e-b3ad-f6fff5d888fa/revoke-keys/0.log" Oct 31 22:21:16.872967 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:16.872892 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7084ad42-jcg6v_71ce4284-2fc4-45e8-954e-596c3d17d73d/revoke-keys/0.log" Oct 31 22:21:17.060758 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:17.060724 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8caf3a09-hxfkd_8c9e5b6e-0f5e-4cbe-ba83-4e12ae4422b4/revoke-keys/0.log" Oct 31 22:21:17.233914 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:17.233883 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-961790d1-vjxh5_8aac996a-26d0-4c19-b7e0-23ff356ebe37/revoke-keys/0.log" Oct 31 22:21:17.421878 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:17.421851 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a8e1165d-wtbhv_0fcd8c4f-3298-4c2d-82a7-325ca715c9ac/revoke-keys/0.log" Oct 31 22:21:17.617410 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:17.617333 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b3612f78-z6zhx_2b61b5aa-913a-40be-900b-244b9706cf03/revoke-keys/0.log" Oct 31 22:21:18.013262 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:18.013237 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-wj5cc_ed38f675-4a9b-440f-8be4-c617ba27fc1d/postgres/0.log" Oct 31 22:21:19.509852 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:19.509818 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c695f6bc5-nbdsf_dbeb690d-72d7-44bc-bc54-e4512acd0d09/authorino/0.log" Oct 31 22:21:19.943458 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:19.943370 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-774gh_9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777/manager/0.log" Oct 31 22:21:20.131996 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:20.131960 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vpn5l_90d377a0-ae28-4f98-9020-54e37535c7e4/kuadrant-console-plugin/0.log" Oct 31 22:21:20.850086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:20.850054 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ltl6h_80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5/manager/0.log" Oct 31 22:21:21.631900 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:21.631862 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28_587a84b8-0ce0-4c87-805d-86e2dfea3eb6/istio-proxy/0.log" Oct 31 22:21:22.028110 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:22.028085 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5587c8dd5c-gs8xq_00a2cecb-5a18-4e86-a966-bc536811d816/kube-auth-proxy/0.log" Oct 31 22:21:22.608229 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:22.608201 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-c7qkl_ddb69f48-47a6-492f-9564-33f725285f09/istio-proxy/0.log" Oct 31 22:21:23.008281 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:23.008249 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b_245bacec-3301-4fc3-afb3-b5795bc24233/istio-proxy/0.log" Oct 31 22:21:31.308976 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:21:31.308940 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:21:35.752908 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.752874 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bgz52/must-gather-44mf6"] Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753223 4294 cpu_manager.go:414] "containerMap: removing container" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753234 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753248 4294 cpu_manager.go:414] "containerMap: removing container" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753253 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753261 4294 cpu_manager.go:414] "containerMap: removing container" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753266 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753286 4294 cpu_manager.go:414] "containerMap: removing container" podUID="7064cc92-7aea-4977-8bb8-a357e08b00ad" containerName="revoke-keys" Oct 31 22:21:35.753339 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753290 4294 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7064cc92-7aea-4977-8bb8-a357e08b00ad" containerName="revoke-keys" Oct 31 22:21:35.753612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753357 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="82abe779-421c-476a-8f64-bdecf9dccf3f" containerName="main" Oct 31 22:21:35.753612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753368 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="7064cc92-7aea-4977-8bb8-a357e08b00ad" containerName="revoke-keys" Oct 31 22:21:35.753612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753376 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5d13372-0bc0-4ffa-886b-d91540606cbe" containerName="main" Oct 31 22:21:35.753612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.753382 4294 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc39eac6-d68f-4420-87d7-48f4c0fbbe96" containerName="istio-proxy" Oct 31 22:21:35.756332 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.756293 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:35.786002 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.785977 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bgz52\"/\"kube-root-ca.crt\"" Oct 31 22:21:35.786125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.786002 4294 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bgz52\"/\"openshift-service-ca.crt\"" Oct 31 22:21:35.786125 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.786082 4294 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bgz52\"/\"default-dockercfg-8gbfk\"" Oct 31 22:21:35.788077 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.788053 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgz52/must-gather-44mf6"] Oct 31 22:21:35.853263 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.853236 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhnjh\" (UniqueName: \"kubernetes.io/projected/b78f6933-b3d2-4766-abe1-8ab253bff041-kube-api-access-jhnjh\") pod \"must-gather-44mf6\" (UID: \"b78f6933-b3d2-4766-abe1-8ab253bff041\") " pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:35.853424 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.853293 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b78f6933-b3d2-4766-abe1-8ab253bff041-must-gather-output\") pod \"must-gather-44mf6\" (UID: \"b78f6933-b3d2-4766-abe1-8ab253bff041\") " pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:35.953760 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.953730 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhnjh\" (UniqueName: \"kubernetes.io/projected/b78f6933-b3d2-4766-abe1-8ab253bff041-kube-api-access-jhnjh\") pod \"must-gather-44mf6\" (UID: \"b78f6933-b3d2-4766-abe1-8ab253bff041\") " pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:35.953912 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.953790 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b78f6933-b3d2-4766-abe1-8ab253bff041-must-gather-output\") pod \"must-gather-44mf6\" (UID: \"b78f6933-b3d2-4766-abe1-8ab253bff041\") " pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:35.954086 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.954070 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b78f6933-b3d2-4766-abe1-8ab253bff041-must-gather-output\") pod \"must-gather-44mf6\" (UID: \"b78f6933-b3d2-4766-abe1-8ab253bff041\") " pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:35.981220 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:35.981195 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhnjh\" (UniqueName: \"kubernetes.io/projected/b78f6933-b3d2-4766-abe1-8ab253bff041-kube-api-access-jhnjh\") pod \"must-gather-44mf6\" (UID: \"b78f6933-b3d2-4766-abe1-8ab253bff041\") " pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:36.070254 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:36.070196 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgz52/must-gather-44mf6" Oct 31 22:21:36.365033 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:36.365007 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgz52/must-gather-44mf6"] Oct 31 22:21:36.371359 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:21:36.371260 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb78f6933_b3d2_4766_abe1_8ab253bff041.slice/crio-2b0c7fea9d849f8cb7f25fe27c8989c1ee0f275851a756f09c24116df3f4787a WatchSource:0}: Error finding container 2b0c7fea9d849f8cb7f25fe27c8989c1ee0f275851a756f09c24116df3f4787a: Status 404 returned error can't find the container with id 2b0c7fea9d849f8cb7f25fe27c8989c1ee0f275851a756f09c24116df3f4787a Oct 31 22:21:36.372829 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:36.372812 4294 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 22:21:36.977131 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:36.977093 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgz52/must-gather-44mf6" event={"ID":"b78f6933-b3d2-4766-abe1-8ab253bff041","Type":"ContainerStarted","Data":"2b0c7fea9d849f8cb7f25fe27c8989c1ee0f275851a756f09c24116df3f4787a"} Oct 31 22:21:37.031172 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:37.031132 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v"] Oct 31 22:21:37.044069 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:37.044037 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7084ad42-jcg6v"] Oct 31 22:21:37.982691 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:37.982652 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71ce4284-2fc4-45e8-954e-596c3d17d73d" path="/var/lib/kubelet/pods/71ce4284-2fc4-45e8-954e-596c3d17d73d/volumes" Oct 31 22:21:37.985814 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:37.985782 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgz52/must-gather-44mf6" event={"ID":"b78f6933-b3d2-4766-abe1-8ab253bff041","Type":"ContainerStarted","Data":"429ff25c3a903b4d2c3fa1ea4b4cdb88eac1e2dab3ecfa2c2d7f0b4489b12583"} Oct 31 22:21:37.985955 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:37.985830 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgz52/must-gather-44mf6" event={"ID":"b78f6933-b3d2-4766-abe1-8ab253bff041","Type":"ContainerStarted","Data":"8d395a7310dd561ea5d59017591609efe6f3eb0404586eb1d2375757e084d3da"} Oct 31 22:21:38.030077 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:38.030029 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bgz52/must-gather-44mf6" podStartSLOduration=2.259595929 podStartE2EDuration="3.030016853s" totalImagesPullingTime="770.420924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 22:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:21:38.02920086 +0000 UTC m=+3938.154308093" watchObservedRunningTime="2026-10-31 22:21:38.030016853 +0000 UTC m=+3938.155124137" Oct 31 22:21:40.434808 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:40.434772 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-29psw_bc8d0364-096a-49f7-87b3-ae72ae71f1a1/global-pull-secret-syncer/0.log" Oct 31 22:21:41.746840 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:41.745997 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qwzj8_751552d9-639f-42f2-bdcb-06fd195d7c2e/konnectivity-agent/0.log" Oct 31 22:21:41.994323 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:41.994037 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-156.ec2.internal_48e3e5e26418af9437f262086f802d3b/haproxy/0.log" Oct 31 22:21:45.359203 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:45.358940 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7m5zl_26251b9d-2632-449f-9e30-eecdf1204060/manager/0.log" Oct 31 22:21:47.399324 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:47.398448 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c695f6bc5-nbdsf_dbeb690d-72d7-44bc-bc54-e4512acd0d09/authorino/0.log" Oct 31 22:21:48.031620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:48.031583 4294 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb"] Oct 31 22:21:48.043504 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:48.043469 4294 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05291d46-k4pdb"] Oct 31 22:21:48.149580 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:48.149548 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-774gh_9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777/manager/0.log" Oct 31 22:21:48.267244 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:48.267214 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vpn5l_90d377a0-ae28-4f98-9020-54e37535c7e4/kuadrant-console-plugin/0.log" Oct 31 22:21:49.247332 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:49.242854 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ltl6h_80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5/manager/0.log" Oct 31 22:21:49.991143 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:49.991105 4294 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec6fd83-e9bd-47e7-8779-e41f41d736e8" path="/var/lib/kubelet/pods/7ec6fd83-e9bd-47e7-8779-e41f41d736e8/volumes" Oct 31 22:21:50.107326 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.103551 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7m5zl_26251b9d-2632-449f-9e30-eecdf1204060/manager/0.log" Oct 31 22:21:50.778744 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.778707 4294 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf"] Oct 31 22:21:50.797697 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.797669 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:50.812062 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.809829 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf"] Oct 31 22:21:50.927905 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.927874 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-podres\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:50.927905 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.927909 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-lib-modules\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:50.928160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.927933 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-sys\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:50.928160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.928076 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-proc\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:50.928160 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:50.928140 4294 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnbd\" (UniqueName: \"kubernetes.io/projected/5deb5d17-9290-449f-b1f8-160b7a4eae69-kube-api-access-hxnbd\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029491 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-podres\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029533 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-lib-modules\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029569 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029569 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-sys\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029611 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-proc\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029634 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-podres\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029656 4294 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnbd\" (UniqueName: \"kubernetes.io/projected/5deb5d17-9290-449f-b1f8-160b7a4eae69-kube-api-access-hxnbd\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029665 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-lib-modules\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029706 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-sys\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.029883 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.029752 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5deb5d17-9290-449f-b1f8-160b7a4eae69-proc\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.054492 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.054449 4294 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnbd\" (UniqueName: \"kubernetes.io/projected/5deb5d17-9290-449f-b1f8-160b7a4eae69-kube-api-access-hxnbd\") pod \"perf-node-gather-daemonset-d7fgf\" (UID: \"5deb5d17-9290-449f-b1f8-160b7a4eae69\") " pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.120500 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.120459 4294 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:51.436463 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:51.436431 4294 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf"] Oct 31 22:21:51.438186 ip-10-0-136-156 kubenswrapper[4294]: W1031 22:21:51.438155 4294 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5deb5d17_9290_449f_b1f8_160b7a4eae69.slice/crio-48616ff74ca4fcfab6cdea42936223cb90ca6ca3a64e3c94aa4dc6338d40b88a WatchSource:0}: Error finding container 48616ff74ca4fcfab6cdea42936223cb90ca6ca3a64e3c94aa4dc6338d40b88a: Status 404 returned error can't find the container with id 48616ff74ca4fcfab6cdea42936223cb90ca6ca3a64e3c94aa4dc6338d40b88a Oct 31 22:21:52.202280 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:52.202245 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" event={"ID":"5deb5d17-9290-449f-b1f8-160b7a4eae69","Type":"ContainerStarted","Data":"7a8367f7d7ef4aa668959931fb9443608eaf670a3ab34e143d7b6910b019e464"} Oct 31 22:21:52.202777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:52.202321 4294 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" event={"ID":"5deb5d17-9290-449f-b1f8-160b7a4eae69","Type":"ContainerStarted","Data":"48616ff74ca4fcfab6cdea42936223cb90ca6ca3a64e3c94aa4dc6338d40b88a"} Oct 31 22:21:52.202777 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:52.202418 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:21:52.248979 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:52.247129 4294 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" podStartSLOduration=2.247109977 podStartE2EDuration="2.247109977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 22:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 22:21:52.245600186 +0000 UTC m=+3952.370707416" watchObservedRunningTime="2026-10-31 22:21:52.247109977 +0000 UTC m=+3952.372217211" Oct 31 22:21:52.291507 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:52.291467 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8474bc4858-594xw_fac108a2-b5b6-4241-82a2-e180d756608a/metrics-server/0.log" Oct 31 22:21:52.924962 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:52.924933 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-plnrb_19000f2f-b5f2-4213-9f97-3928d2880662/node-exporter/0.log" Oct 31 22:21:53.040434 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:53.040408 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-plnrb_19000f2f-b5f2-4213-9f97-3928d2880662/kube-rbac-proxy/0.log" Oct 31 22:21:53.157695 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:53.157670 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-plnrb_19000f2f-b5f2-4213-9f97-3928d2880662/init-textfile/0.log" Oct 31 22:21:54.073537 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.073501 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddb09a2f-1e81-4837-937e-f50442ec2d03/prometheus/0.log" Oct 31 22:21:54.190966 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.190931 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddb09a2f-1e81-4837-937e-f50442ec2d03/config-reloader/0.log" Oct 31 22:21:54.311108 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.311064 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddb09a2f-1e81-4837-937e-f50442ec2d03/thanos-sidecar/0.log" Oct 31 22:21:54.422443 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.422367 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-7m5zl_26251b9d-2632-449f-9e30-eecdf1204060/manager/0.log" Oct 31 22:21:54.428287 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.428264 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddb09a2f-1e81-4837-937e-f50442ec2d03/kube-rbac-proxy-web/0.log" Oct 31 22:21:54.545374 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.545346 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddb09a2f-1e81-4837-937e-f50442ec2d03/kube-rbac-proxy/0.log" Oct 31 22:21:54.662839 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.662799 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddb09a2f-1e81-4837-937e-f50442ec2d03/kube-rbac-proxy-thanos/0.log" Oct 31 22:21:54.780788 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:54.780764 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_ddb09a2f-1e81-4837-937e-f50442ec2d03/init-config-reloader/0.log" Oct 31 22:21:55.155202 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:55.155130 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-f945h_ed8fd589-3f8a-4ede-9a1b-2749f7a98395/prometheus-operator-admission-webhook/0.log" Oct 31 22:21:55.677279 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:55.677240 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555f8d6d4b-prhz2_26b055db-6639-407e-a248-c1339b966438/thanos-query/0.log" Oct 31 22:21:55.794258 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:55.794226 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555f8d6d4b-prhz2_26b055db-6639-407e-a248-c1339b966438/kube-rbac-proxy-web/0.log" Oct 31 22:21:55.911722 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:55.911689 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555f8d6d4b-prhz2_26b055db-6639-407e-a248-c1339b966438/kube-rbac-proxy/0.log" Oct 31 22:21:56.036808 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:56.036774 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555f8d6d4b-prhz2_26b055db-6639-407e-a248-c1339b966438/prom-label-proxy/0.log" Oct 31 22:21:56.183933 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:56.183903 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555f8d6d4b-prhz2_26b055db-6639-407e-a248-c1339b966438/kube-rbac-proxy-rules/0.log" Oct 31 22:21:56.301197 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:56.301125 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-555f8d6d4b-prhz2_26b055db-6639-407e-a248-c1339b966438/kube-rbac-proxy-metrics/0.log" Oct 31 22:21:58.008257 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:58.008220 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28_587a84b8-0ce0-4c87-805d-86e2dfea3eb6/istio-proxy/0.log" Oct 31 22:21:58.219620 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:21:58.219595 4294 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bgz52/perf-node-gather-daemonset-d7fgf" Oct 31 22:22:01.313357 ip-10-0-136-156 kubenswrapper[4294]: E1031 22:22:01.313317 4294 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vx47h" containerName="extractor" Oct 31 22:22:03.085959 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:03.085930 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5587c8dd5c-gs8xq_00a2cecb-5a18-4e86-a966-bc536811d816/kube-auth-proxy/0.log" Oct 31 22:22:03.283923 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:03.283892 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-c7qkl_ddb69f48-47a6-492f-9564-33f725285f09/istio-proxy/0.log" Oct 31 22:22:03.405332 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:03.405229 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-856b6dffcf-vdw7b_245bacec-3301-4fc3-afb3-b5795bc24233/istio-proxy/0.log" Oct 31 22:22:03.978164 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:03.978132 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28_587a84b8-0ce0-4c87-805d-86e2dfea3eb6/istio-proxy/0.log" Oct 31 22:22:04.002743 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:04.002705 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k6sw4_72e88409-4f64-46c1-8818-d625e4411683/dns/0.log" Oct 31 22:22:04.119389 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:04.119363 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k6sw4_72e88409-4f64-46c1-8818-d625e4411683/kube-rbac-proxy/0.log" Oct 31 22:22:04.456353 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:04.456321 4294 scope.go:122] "RemoveContainer" containerID="bc16856dd3160949babdf9d09a88b6e97e85fea72519fc46d0e34eda5d69bf88" Oct 31 22:22:04.465612 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:04.465591 4294 scope.go:122] "RemoveContainer" containerID="072b88d58851405dbf7340fc4d288fddbc62134511d93cb82aaeb2472f81a2cd" Oct 31 22:22:04.473990 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:04.473968 4294 scope.go:122] "RemoveContainer" containerID="cc930a4ceace553996abc7d76161f6b1c0987652e7381f265fd83edea6a11ac2" Oct 31 22:22:04.502188 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:04.502165 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-74gq6_bbfa483f-5315-4500-8f1b-e42d50e9b31b/dns-node-resolver/0.log" Oct 31 22:22:05.912874 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:05.912840 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-c59468d77-4655d_c7418585-f650-4670-87ba-241b6752a329/registry/0.log" Oct 31 22:22:06.031553 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:06.031519 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2v76n_4af55054-ae66-45a2-a8ea-4a22b5e95bfe/node-ca/0.log" Oct 31 22:22:07.028387 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:07.028336 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c695f6bc5-nbdsf_dbeb690d-72d7-44bc-bc54-e4512acd0d09/authorino/0.log" Oct 31 22:22:07.119935 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:07.119897 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-774gh_9f8bd849-a9b6-4f4d-83c6-0bbb9ee0f777/manager/0.log" Oct 31 22:22:07.177715 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:07.177675 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vpn5l_90d377a0-ae28-4f98-9020-54e37535c7e4/kuadrant-console-plugin/0.log" Oct 31 22:22:07.580644 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:07.580608 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-ltl6h_80e8eb5b-ed9f-4abf-998f-3a92d0a8cca5/manager/0.log" Oct 31 22:22:08.145051 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:08.145020 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-59cf4dc6bfstm28_587a84b8-0ce0-4c87-805d-86e2dfea3eb6/istio-proxy/0.log" Oct 31 22:22:09.465603 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:09.465559 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5587c8dd5c-gs8xq_00a2cecb-5a18-4e86-a966-bc536811d816/kube-auth-proxy/0.log" Oct 31 22:22:09.672973 ip-10-0-136-156 kubenswrapper[4294]: I1031 22:22:09.672938 4294 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-c7qkl_ddb69f48-47a6-492f-9564-33f725285f09/istio-proxy/0.log"